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Preface 



"1 Evaluating educational technology programs can be a challenging endeavor. 

|The j. j As more and more states, districts, and schools develop technology plans to 

J CHALLENGE j ensure that technology will be used effectively to benefit student learning and 
achievement, the need to understand technology’s impact on improving 
student achievement will become even greater. Furthermore, the funding that 
states and districts receive to implement components of technology plans often 
requires some type of evaluation. The question thus becomes, how do you 
evaluate educational technology programs that vary in: 



• the types of students they serve 

• the curriculum areas in which teachers use technology, and 

• the type of technology itself? 



The Guide 



This guide was developed for the U.S. Department of Education by the 
American Institutes for Research in conjunction with its formative evaluation of 
the Technology Literacy Challenge Fund. The guide represents a joint effort 
among the Office of Educational Research and Improvement, the Office of 
Educational Technology, and the Office of Elementary and Secondary 
Education. The guide should be viewed as a tool for individuals who have little 
or no formal training in research or evaluation. Its intended informal style and 
accompanying worksheets provide the basic principles of evaluation and are 
designed to help district and school personnel gain an overview of and ideas 
for evaluating local technology initiatives. 



The guide is not meant to be the key to conducting a perfect evaluation. There 
is no such thing as a perfect program and no such thing as a perfect 
evaluation. Rather, the goal of this handbook is to provide educators a 
resource with which to jump into the evaluation process, learning as they go. 
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Overview of Handbook 



IT* 







Purpose of 
Guide 



Your school has just received a grant to purchase new computers and to train 
teachers to integrate technology into their teaching - activities that will enable 
the school to achieve some of the goals outlined in both the district and 
school technology plans. Everyone is excited about the new technology and 
the possibilities it has for improving teaching and learning. 



However, people have questions. The agency funding the grant wants to 
know if it is getting its "money's worth"; the superintendent wants to know if 
teachers are using technology in their instruction and whether the technology 
is affecting student work and motivation; parents want to know if the new 
technology is improving standardized test scores; and the district professional 
development coordinator wants to know the effect of training on teacher and 
student technology skills. Someone has to try to answer all of these 
questions. For better or worse, this someone is you! 

Yes, there are strings attached to the grant. In this age of limited resources, 
people want to know if the money they are spending is producing results, and 
they want answers soon. Yes, your time is limited. You are still trying to 
coordinate the purchase of the new equipment and make sure that it is 
running properly. You are also trying to make sure that teachers are getting 
the training they need so that the technology will be used appropriately. And 
on top of all of this, you must set up an evaluation that will satisfy many 
different interests. So the question becomes: How do I do this evaluation and 
how can I make it useful to me as well as everybody else? 

This guide is designed to help you through the evaluation process and to 
make it as painless as possible. Primarily geared to educators or 
administrators with little or no research experience, it is intended for use at 
the district or school level. 



Basic 

Organizing 

Questions 



As you begin the evaluation process, you will have a number of questions. 
This guide is organized in sections by typical questions one might ask in 
going through the evaluation process. Within each section, a number of 
materials will guide you in answering questions for your particular program. 
Materials will consist of tips and “tricks of the trade" along with worksheets 
and ideas for additional resources. The following questions are addressed: 



Why am I evaluating? 

What is an evaluation anyway? 
Where do I start? 

What questions should I ask? 
What information do I need to 
collect? 



What's the best way to collect my 
information? 

What are my conclusions? 

How do I communicate my results? 
Where do I go from here? 



In addition, throughout this guide, you will have the opportunity to follow a 
sample school district (Rivers) through the evaluation process. It is hoped 
that this example will give you "real life" ideas for your own evaluation. 

So, let's get started! On the next page is an overview of “Rivers." 
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I Rivers Overview 




Introduction to Rivers School District* 




Rivers School District serves 6,000 primarily working-class students just outside of a large 
metropolitan city. The district consists of one high school, two middle schools, and four 
elementary schools. The district has grown fairly steadily over the years, with much of the growth 
coming from a large minority population moving out of the city. 

The school district has many issues to deal with: 

• The attendance rate has dropped from 95 to 85 percent. 

• The dropout rate has been climbing slowly but steadily. 

• Standardized test scores have remained consistently below the state average for the past 
decade, with reading scores exceptionally low. 

Also, Rivers has only recently jumped onto the "technology bandwagon." Two years ago, 
administrators formed a technology planning committee to determine how technology could be 
used to improve student learning. The committee finished devising a technology plan last year 
that outlined two ways in which technology could help alleviate some of the problems at Rivers. 
First, learning centers could be set up in the libraries in the elementary schools that could be used 
not only during the school day but could also be open after school hours for students to do 
homework and receive extra tutoring. In addition, as only 10 percent of the teachers had Internet 
accounts and only 5 percent of them had used technology in their classes, the committee felt that 
professional development was an absolute necessity. 

Just after the committee released the technology plan, the district superintendent learned that 
funds were available for educational technology through the Technology Literacy Challenge Fund, 
a Federal program that assists states to implement their technology plans. The committee wrote 
a proposal that incorporated the activities above and received $200,000 in funding. 

The $200,000 in funding was used for the two main activities outlined in the technology plan. First, 
media centers with Internet connections were set up in the libraries of the elementary schools. 
Second, summer training was offered to all teachers in the district at two levels. There was a 
course on basic computer applications and computer use as well as a course on designing lessons 
incorporating technology. Each course lasted approximately two weeks. 

As part of some additional state funding for technology, Kathy Wilson, a teacher, was appointed 
the district “technology coordinator” and charged with overseeing the implementation of grant 
activities. In addition, Kathy was given the responsibility of evaluating the program. As a teacher, 
not a researcher, Kathy has a lot of questions about how to evaluate a new program. 
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Am 0 Evaluating? 




There are many different reasons to evaluate a program. Many people 
think of program evaluation as a nerve-wracking process meant to determine 
continued funding or recognition. Although making decisions on continued 
funding or recognition can be a purpose of program evaluation, there are 
many other, less worrisome reasons to evaluate a program. Some of these 
follow: 

• To provide information to program personnel and others on aspects of 
the program that work well and potential problems. 

• To catch potential problems early in the program so they can be 
corrected before more serious problems occur. 

• To guide further evaluation efforts. For instance, an evaluation may bring 
to light issues that need to be examined in greater detail or an initial 
evaluation of program implementation may be used, in part, to guide a 
later evaluation of long-term impact. 

• To provide information on what technical assistance may be needed. 

• To determine what impact the program is having on participants. 

In general, one important purpose of any evaluation should be continual 
program improvement. By using evaluation results, program administrators 
can better understand how their program is working and where it is headed. 
With this greater understanding, they are better able to make decisions that 
will improve the program in the long run. 

The purpose of evaluation should influence the design of the evaluation. 
Evaluation can take many different forms. One factor influencing the form an 
evaluation could take is the purpose for which the evaluation was designed in 
the first place. As an example, look at the following scenarios: 

• School A has recently offered a series of professional development 
activities for teachers on different computer applications. To plan for 
additional professional development, the school wants to evaluate the 
current abilities of teachers who participated as well as the needs of 
teachers who did not. Thus, data will need to be collected from a wide 
range of teachers. 

• School B is planning a series of professional development activities for 
teachers on different computer applications. This school wants to use 
evaluation to decide if the professional development was a success and if 
it should be repeated. In this case, data will probably be collected only 
from teachers who participated in the program and will be collected both 
before and after participation. 

One evaluation can also have multiple purposes. For instance, an evaluation 
of student computer use could be designed both to assess the types of 
programs students prefer and to inform those designing a new curriculum on 
the topics needing attention in computer literacy courses. 

These are different examples of why others may be evaluating. Why are you 
evaluating? 



Purposes for 
Evaluation 
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Rivers 






Why Is Rivers Evaluating? 

One of the first things that Kathy Wilson, the technology coordinator at Rivers, asked herself when 
she was given the task of evaluating the technology initiative, was why the evaluation was 
necessary in the first place. 

Kathy realized that different stakeholders may have different reasons for wanting an evaluation, 
and that it would be a good idea to take others’ needs and interests into account when planning 
the evaluation. So, to better understand the overall purpose for the district’s efforts to evaluate its 
technology plan, Kathy sat down and listed each of the major stakeholders and what they wanted 
to know about the effort. She came up with the following. 

Stakeholder y Whxxtjhey want' from/ an/ evaluation/ 

M r.Joney, To find out if the/technology andtyatntrvg'are/helping^to 

HS Principal/ alleviate/ }ccme/ of the/prohleeny at the/ ydvool (e/.g'., low 

reading' yoorey) 



Vr. Walker, Same/ a# Mr. Jcmey, butal^waAdyto-findoutifitiymore/ 

Superintendent coyt- effective/ than other optiony 



M y. Phillip y 
Professional/ 
“Develop <ment 



Toget information/that will/help her plan and/ improve/ 
nest year’ y p rofessional/ development activities' 



Teachery To- determine' Of ytxident reading' skilly are' improving' ay ou 

result of Internet-bated/ lemony and/ availability of the/ 

media/ centery 



Parenty 



Toknow if the/ technology iy making' their children/ more/ 
computer literate/ and if it will/ give/ their children/ greater 
opportunities' in/ the/ future/ 



From this analysis, Kathy concluded that most people wanted an evaluation to provide information 
on how specific efforts to implement the technology plan are, at the present time, “working." Since 
she knows that she will probably be responsible for implementing further technology efforts in the 
future, Kathy also decided that, for her own use, she wants information on the implementation 
process. She feels that detailed information on what it took to get the program going will help her 
to better understand both the current state of the program and future program outcomes. So, 
Kathy's answer to the question of "Why am I doing this evaluation?" has two major parts: 



• To find out if the program is beginning to produce desired results (e.g., reading test scores are 
improving, professional development is effective, the program is cost effective) 

• To obtain information on implementing the program 
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"Why Am 1 Evaluating" Worksheet ;■ 

To answer the larger question of "Why am 1 evaluating?" you will need to consider the different 
stakeholders in your program and what they want out of the evaluation. 

Stakeholders What they want from an evaluation 








What do you want from an evaluation? 






Looking at what both you and other stakeholders want from an evaluation, what are two or three 
major reasons or purposes for doing your evaluation? 






How might these reasons affect the design of your evaluation? 
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What Is an Evaluation Anyway? 



P ; . - - r~;“ X k*W "^r*T*r ^T~ 

{ . •■••• ‘ x , • ‘<k { ■ > r ;.t . -llh' %-i <M*- 'f? : -'I? -’ i 

DEFINITION Rossi and Freeman (1993) define evaluation as "the systematic application of 
social research procedures for assessing the conceptualization, design, 
implementation, and utility of ... programs." There are many other similar 
definitions and explanations of "what evaluation is" in the literature. Our view is 
that, although each definition, and in fact, each evaluation is slightly different, 
there are several different steps that are usually followed in any evaluation. It is 
these steps which guide the questions organizing this handbook. An overview of 
the steps of a "typical" evaluation follows. 



Evaluation j 
Overview 1 
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Formative 

Evaluation 



Evaluators often talk about two different types of evaluations: "formative" and 
"summative." A formative evaluation is usually conducted in the early stages of 
a program and addresses questions about implementation and ongoing 
planning. This type of evaluation typically examines process rather than product. 
In the case of technology programs, a formative evaluation might ask if 
equipment was received on time, if the expected number of teachers received 
professional development, what skills teachers learned from professional 
development, or if the equipment is being used in the classroom as planned. 

Formative evaluations are useful for various purposes. For example: 



Formative 

Evaluation 

Examples 



• They may help catch problems early on, while they can still be corrected. 

• They are an evaluation of process, so they may be useful in understanding 
why different outcomes emerge and improving program management. 

• They provide an opportunity to collect baseline data for future summative (or 
“impact”) evaluations. 

• They help identify appropriate outcomes for summative evaluations. 

The following are some examples of formative evaluation: 

Example A - "Why didn't they show up?" At the beginning of the school year, 
Smalltown High School decided to provide the first part of a two-part teacher 
professional development series on the integration of technology into the 
classroom. While the teachers attending the professional development were 
satisfied and seemed to benefit from it, the turnout was lower than expected. 
Upon further investigation, the evaluators found out that, since the professional 
development was offered after school, many teachers who were involved in 
leading extra-curricular activities could not attend. Learning this, evaluators 
surveyed teachers about their preferences and availability. From the survey 
results, the decision was made to have both Saturday training and training during 
the summer. 



Example B - "Things don't always go as planned. " During the summer, Big 
City Elementary completed all its wiring. The school’s plan was to install 
equipment in the fall and provide professional development for teachers that 
would teach them how to use the new equipment in the spring. In December, as 
part of its overall evaluation plan, the school assessed whether the equipment 
had been installed as intended. It turned out that most of the computer 
equipment was backordered and would not be installed in time for the 
professional development. Evaluators determined that, because of the delay, 
prices had dropped and better equipment could be obtained at a similar cost. 
The school quickly updated specifications and was able not only to receive the 
equipment in time for the spring professional development, but to purchase 
better equipment than originally intended. 






\n Educator's Guide to Evaluating the Use of Technology in Schools and Classrooms 



Page 7 



jSUMMAUVE 

j Evaluation 



SUMMATIVE 

Evaluation 

Examples 



Summative evaluations assess program outcomes or impacts. To determine the 
relationship of different factors to outcomes, similar to formative evaluations, 
some information used in summative evaluations is collected early in the life of a 
program (e.g., baseline data, test scores). Unlike formative evaluations, 
however, a portion of the information is collected after the program has been 
completely implemented and adequate time has passed to expect outcomes to 
occur. In terms of evaluating educational technology, a summative evaluation 
might ask if teacher technology skills improved as a result of a professional 
development activity, if teachers are using technology to a greater extent in their 
instruction, or if technology improved student motivation or performance. 

Some advantages of summative evaluations include: 

• They can, if designed correctly, provide evidence for a cause-and-effect 
relationship. 

• They assess long-term effects. 

• They provide data on impacts. 

• They can provide data on change across time. 

Examples of summative evaluation follow: 

Example A - "Which type of professional development is better?" 
Administrators at Smalltown High had two different approaches to providing 
professional development to teachers on the use of technology in their 
classrooms. One group felt that, since many teachers did not have extensive 
computer knowledge, it was important to teach them how to use the software 
before teaching them how to integrate the software into instruction. This group 
proposed a six-week schedule (one session per week), with three sessions on 
the software itself and three sessions on integrating it into teaching. Another 
group felt that teachers would learn how to use the software as a result of it 
simply being available, so this group proposed a full six sessions focusing solely 
on curriculum integration. The school decided to test both options. Teachers 
signed up for professional development and then were randomly assigned to one 
of the two types. One year after participating, evaluators surveyed the teachers 
on the use of different types of software in their instruction and observed 
teachers’ use of technology in the classroom. 

Example B - "Test scores, test scores. " One year after spending a large 
amount of money on technology, parents of students at Big City Elementary 
started asking whether the new technology would improve test scores. As the 
school planned on adding new technology grade-by-grade, a decision was made 
to follow cohorts of students over time to assess changes in math and reading 
scores. Specifically, the school was interested in comparing the class of 
students going through the school two years before the technology was 
implemented to the class two years after the technology was implemented. To 
assess long-term impacts, the school would monitor test scores of the students 
through graduation from high school. 
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Where Do I Start? 



Starting 

Early 



Planning an evaluation starts with planning for technology. It is best to 
plan your evaluation at the same time that the technology program itself is 
being planned. Early involvement will help you for two reasons. First, it will 
allow you to collect baseline data. Second, it will allow you to structure 
professional development and/or purchases to the evaluation design. For 
instance, in Example A above, teachers were randomly assigned to different 
kinds of professional development so that different types of teachers (e.g., 
novice vs. veteran) were distributed roughly equally among both types. To 
implement random assignment, the evaluator had to be involved in program 
planning. 

What if the program is already underway? Is it too late? No, it's not too 
late. Though you probably won't have as many options for structuring the 
evaluation, you can still start a good evaluation after the program has begun. 



Helpful 

Documents 



Ok, so where do I start? Start by learning as much as you can about the 
background and goals of the program. You can do this by reviewing a 
number of different documents: 



Talk to 
People 



I Other 
Resources 



• Technology plans. Technology plans will give you the history and 
intended uses of technology in the area. They will help you understand 
how plans have changed over the years, where people believe technology 
should be at different points in time, previous data collection activities, and 
what people believe will be the long-term benefits of technology. 

• Information from previous evaluations. Was information collected prior to 
implementing the program? Was a "needs assessment" done? Were 
recommendations followed? It will be worth looking at this information so 
that you do not duplicate efforts and can learn from previous experiences. 

• Curriculum and/or standards. Does the state, district, or school have a set 
curriculum or standards? Is technology a separate topic or is it integrated 
into different content areas? Looking at these documents will give you 
information on how technology is presently being infused into teaching. 

• School board minutes. Reviewing school board meeting minutes may 
give you an idea of board member plans for technology use in schools. 

In addition to reviewing documents, you will also want to speak with people 
who were involved in planning the program. They will be able to tell you who 
the major stakeholders are as well as the motivation behind the program. 

Finally, you may want to look at other evaluations. Have any previous 
surveys been done at your school or district? Are there similar schools or 
districts doing a technology evaluation? You should talk to persons who 
planned the program, call your state technology coordinator to find out if other 
districts in your state are doing evaluations, and use the Internet to see what 
other states and districts are doing. Indeed, technology itself is a wonderful 
time-saving tool when designing and conducting evaluations. Many associ- 
ations, states, and school districts have placed evaluation designs and 
examples of surveys on the Internet. An example of a good place to start is 
the U.S. Department of Education’s website (http://www.ed.gov/Technology). 
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Rivers 



Where Did Rivers Start? 

The program was already underway before Kathy was asked to do the evaluation, so she was not 
able to provide input into planning. In addition, the district had not collected any technology 
information to date. Thus, her next step was to assess the different pieces of documentation and 
the human resources available to help her devise an evaluation strategy. The following is a list of 
the documents she was able to obtain: 

'Document Commcnty 

State/ Technology Plan/ 



VC^Cct Technology 
Plan 



C urriculum/Standards 



After reviewing documents, Kathy talked to a few people to get more information. First, she called 
the state technology coordinator to get some background on the state plan and information on 
which other districts in the state may be evaluating technology programs. She found out that the 
state plan was written mostly by the technology coordinator with the input of a technology 
committee. She also found out that a technology coordinator from a district about two hours away 
had also recently called requesting evaluation guidance. Kathy got this person’s number, called 
him, and they decided to share information with one another and to meet regularly. 

Finally, Kathy talked with the principal, the superintendent, and the professional development 
coordinator. They gave her a brief overview of district efforts and pointed her to some printed 
resources. She then went to the library in the district office to look up these resources and look for 
additional information on other district evaluations. Using the Internet and a CD-ROM database, 
she found many examples she could draw on, including sample surveys. 



Plan developed/ two y ear y ago- by ytate/ tech coordinator 
Committee/ formed/ (teacher y, techpeople/, school board/) 
Major tabby - wining', hardware/, andprofesbionab 

development 

Wining' to be/ completed/ On two- year y (NOW!) 

Coal/ is to abe/ technology in new media centeny to get 
btudenty connected to the/ world 



Writtenlayt year by committee/ of teacher y 
Plan to have wining- done/ this year, hardware/ next yean 
focay on elementary bchooly firyt, then middle/, then/high 
Coat iy to abe- technology to improve/ reading/ achievement 

State/ ytandardy and curriculum/ are bomewhat different 
from the district curriculum. The ytate curriculum 
conyisty of a/ bet of skills That studenty are expected to 
mayten: among/thebe skills aretechnology competencies. 
The district curriculum doey not mention technology 
much, but when it doey it iy in the content of sample 
lesbonplans or activities to foyter a concept (eg-., 
incorporates using- comp uteny for writing- in English and/ 
for conducting- a science fain project). 
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; ■ "Where Do 1 Start? " Worksheet i ; { 


What documents are available (e.g., state and/or local technology plans)? 




Ask questions about the plans. What major stakeholders were involved in creating the 
documents that were collected? Why were the plans written? Are the state and local plans similar 
(e.g., have similar timeframes)? Are the plans being implemented? How have plans changed over 
the years? What are some major goals in the plans (e.g., "use technology to improve student 
writing skills”)? 








How is technology integrated into the curriculum and/or standards? 




Whom could you talk to about the evaluation? 




Who else may be collecting data you can use? 


Are there any other resources that you learned about by talking to people? 




What does the Internet have to offer? 
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What Questsoinis Should I Ask? 

PROGRAM j First, review program goals. Figuring out what questions to ask is probably 

GOALS ' one of the most difficult tasks in designing an evaluation. One way to begin to 

think of questions is to review program goals. For our purposes here, a goal 
is defined as an “overall objective of a program or initiative.” 



Your 

Priorities 



ji Others' 

' Priorities 



Goals tend to be broad and general. They reflect the overall intent of a 
program or a shared vision of what the program is supposed to accomplish. 
For example, a district’s technology plan could state that all teachers are to 
receive adequate professional development so that they can integrate tech- 
nology into their teaching and help improve student performance. Technology 
goals will differ slightly from district to district and perhaps school to school. 
They are generally reflective, at least in part, of local concerns, state plans or 
goals related to educational technology, other educational reform initiatives, 
and possibly the national technology goals. Reviewing the specific purposes 
that you and others might have for evaluation discussed on the worksheet on 
page 5 and reviewing written plans and documents are two places you might 
start in trying to find information on the goals of your program. 

Next, figure out what you would like to know about the program. Trying 
to answer the question "What questions should I ask?" can begin with asking 
yourself what you want to know or what you would like to find out about your 
technology program. Do you want to know what teachers thought of the train- 
ing? Do you want to know how often students are using the new equipment? 
Do you want to know the extent of student technology skills? What goals do 
you want to see accomplished? What questions do you want answered? 

Then figure out what others would like to know about the program as 
well. Fortunately or unfortunately, your opinion is not the only one that needs 
to be considered. What goals do other people have for the program? What 
questions do others want answered? Some stakeholders that may need to be 
considered include: 



• Funding agency (e.g., Department of Education, foundation, state agency) 

• State officials 

• School boards 

• Legislators 

• School administrators (e.g., principal, superintendent) 

• Teachers 

• Parents 

You can find out what others want to know in two ways. First, you could 
simply ask questions. Ask the person who gave you the responsibility for 
conducting the evaluation about persons who were involved in decision- 
making for the program. He or she should have at least a fairly good idea of 
who the major stakeholders are. In addition, you could get some information 
on individual goals and questions that people want answered from reading 
various planning documents. For instance, if the technology plan includes a 
list of major program goals, then it is a fair assumption that people will want to 
know if these major goals have been or are in the process of being achieved. 
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Other 

Requirements 



Also, keep in mind any other external requirements. Grant money often 
comes with certain “strings” attached. In particular, many grants come with 
legal or contractual requirements. For example, in the case of the 
Technology Literacy Challenge Fund awards, funds are to be targeted to 
areas with the greatest poverty and the greatest need for technology. Some 
types of requirements that you might run into include: 



Thinking 

About 

Outcomes 



• Requirements on what the money can or cannot be used to purchase 

• Requirements on how much money can be used for equipment 
a Requirements for matching funds 

o Requirements on certain types of reports or data that must be provided to 
the district or state (e.g., progress reports) 

Finally, think about how you may want to ask questions about 
outcomes. One of the most pressing questions you are likely to encounter in 
evaluating a technology initiative is whether the initiative produces 
measurable outcomes. For example, you may be asked whether student 
achievement has improved or you may be asked about other potential long- 
term impacts such as changes in classroom teaching methods. You will need 
to begin to think about how you will answer such questions as: 



• Does the availability of technology in the classroom increase the quality 
and/or quantity of student work? 

• Has professional development related to technology changed the way that 
teachers are organizing instruction? Are teachers teaching any differently 
than they did before participating in professional development? 

• Does having access to the Internet increase student knowledge of other 
cultures and traditions? 

• Does the infusion of technology into the curriculum help increase student 
performance on standardized tests? 

In the hustle and bustle of trying to get a technology initiative off the ground, it 
may be tempting to put these types of questions low on your priority list and 
not think about them. On the other hand, you may feel the pressure of 
wanting to know the answers to these types of questions quickly. Neither of 
these approaches is helpful. In general, questions that measure a program’s 
impact take time to answer. You may be more interested in getting your 
program off the ground (e.g., determining whether the basic features of your 
program are in place, whether the computers you were supposed to have 
operating in classrooms are actually there) than thinking about longer-term 
impacts. But don’t forget: it is often these questions about impact that are of 
primary interest to policymakers. 
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Rivers 



WhatQuestions Did Rivers Ask? 



Kathy had already asked herself why different stakeholders wanted an evaluation and what they 
wanted from an evaluation. She had also come up with two major reasons why she wanted an 
evaluation: 

® To find out if the program is beginning to produce desired results (e.g., student reading 
scores are beginning to improve, professional development is effective, the program is cost 
effective) 

® To get information on the implementation process 

Well, these reasons seemed very broad, so Kathy, using the general ideas and the list of 
specific things that others wanted from the evaluation, came up with general goals that people 
were interested in and, within these goals, more specific questions she wanted answered. The 
following is the result of her work: 



| (Jot lb 


Question/ 


Improved/ 

reading/ 

performance/ 


Will the/ new technology and/tratning/help 
improve/ reading/ performance/? 


Improved/ 
dropout and/ 
attendance/ ratey 


Will the/ new technology and/ training/ help lower 
the/ dropout route/ and/ increase/ attendance/? 


Cost-effectiveneyy 


W ill thiy program/ be/ ay or more/ coyt- effective/ 
than other programs that may ihow similar 
result!/? 


Effective/ 

professional 

development 


Will the/ professional develop ment help teacher y 
integrate/ technology into their teaching -? 


Increased 
computer literacy 


Will ytudenty and/ teacher y become/ more/ computer 
literate/ ay a/ result of the/ program/? 



After coming up with the questions above, Kathy reviewed them to determine if there was 
anything she was leaving out. She assessed her priorities and made sure they were included. 
She ran through her list of stakeholders - principal, superintendent, professional development 
coordinator, teachers, parents, herself. All stakeholders were accounted for. 

Finally, Kathy looked at the grant to see if there were any requirements she needed to assess. 
The grant was through the Technology Literacy Challenge Fund, so it was fairly flexible and 
many of the requirements were at the state level. Basically, one of the biggest requirements 
was that the district was responsible for reporting back to the state yearly on what they had 
done. In Kathy's opinion, assessing the goals that she had developed would satisfy the state. 
She would call her state technology coordinator to be sure, but it looked fine and she thought 
she was ready to further define her evaluation plan. 
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"What Questions Should 1 Ask?" Worksheet 



What do you want to know? (You may want to look back to the worksheet on page 5 for ideas on 
why you are evaluating and what you want from an evaluation.) 



What will others want to know (e.g., principal, superintendent, board members, funding agency, 
teachers, parents, students)? 



What other requirements are there? 



After this exercise, what are your "top 5" or "top 10" goals and questions? 

Goals Research Questions to Address Goals 
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Wmr Information Do fl Need to Collect? 



Now that you’ve identified goals and formulated research questions, the next 
task is to figure out what information you will need to address them. In 
evaluator’s terms, you will need to define more clearly your indicators and 
measures. This section is designed to guide you through the process of 
figuring out what kind of information you need to collect. 



Evidence 



To meet requirements and to address questions, you will need evidence. 
Think about the questions that you and others would like to have answered 
about your technology program. First, what evidence would you need to have 
to answer your questions? What evidence will others be looking for? 
Examples of evidence might include: 



• Improvement (or lack of improvement) in test scores 

• Information on quality and quantity of professional development 

• Data on numbers of computers and Internet connections 

® Information on student use (e.g., types, amount, what technology is used 
for and in what subjects) 



Types of 
Information ! 



The type of information needed can vary from study to study. Although 
there are many variations, the type of information you will collect may be 
qualitative or quantitative and either may be appropriate. An example of 
strictly qualitative information would be a narrative description of a program or 
descriptions of how students are using computers. An example of strictly 
quantitative information would be counts of computers in school buildings, 
number of professional development hours, or number of hours students 
spend using computers. Qualitative and quantitative data can reinforce one 
another. For example, knowing both what students do when using 
computers, a qualitative measure, and how much time students spend using 
computers, a quantitative measure, provides a more complete understanding 
of student computer usage than either type of data individually. 



Going From 
Questions to 
Evidence 



Ok, I have my goals and questions and I know there are different kinds 
of information out there - How do I know what information I should get? 
What you want to do is go from the goals and questions you have already 
thought about to "indicators," to "benchmarks,” and then to "measures." Some 
of these terms you may have heard before (e.g., goals), others you may not 
have. Since they are used in a variety of ways by different researchers, it’s 
easy to get lost in the language. For simplicity, we’ll take each of the steps 
one at a time, defining how we use the terms as we go, beginning on the next 
page with an explanation of goals. 
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Goals 



First, review your goals. In the last section, you devised a list of goals and 
research questions. Although goals can vary greatly, they can be thought of 
as generally falling into two broad categories: “intermediate” or “outcome.” 



Intermediate goals are the products or processes necessary to achieve 
outcomes. In one sense, intermediate goals can be described as a “means to 
an end.” The national technology goals, as follows, are good examples of 
intermediate goals. 

• All teachers in the nation will have the training and support they need to 
help students learn using computers and the information superhighway; 

• All teachers and students will have modern multimedia computers in their 
classrooms; 

• Every classroom will be connected to the information superhighway; and 

• Effective and engaging software and online resources will be an integral 
part of every school’s curriculum. 

Intermediate goals such as these - and the indicators, benchmarks, and 
measures developed from them - are important for determining whether a 
program is up and running as it was intended and are vital for providing 
possible explanations for ultimate outcomes. 

Outcome goals reflect the final product, or “outcome,” desired. In many 
cases, in educational research, outcome goals are related broadly to 
improved teaching and learning. For example, one may want to know if 
access to word processing software will improve student writing. In this case, 
access to word processing software would be an intermediate goal and 
improvement in student writing would be an outcome goal. 

The relationship between intermediate goals and outcome goals is not always 
straightforward. In some cases, intermediate results (e.g., more computers in 
the classroom, added professional development) work together to enable 
desired outcomes to occur. In other cases, one intermediate result will 
influence an outcome by way of another intermediate result (e.g., increased 
professional development leads to increased integration of technology into the 
curriculum, which then leads to higher student performance). 

The ultimate goal of many technology programs is to increase various aspects 
of student achievement and, in some cases, teacher performance. For 
instance, a district may want to use technology to improve the quality of 
student writing. However, there are steps that must occur before determining 
whether the use of technology has improved student writing. There must be 
(a) enough technology available to make a difference, (b) teachers 
adequately trained to use that technology, and (c) integration of technology 
into the curriculum and classroom activities. In this case, the outcome goal is 
improved student writing and the intermediate goals, those “means to an 
end,” are the availability of technology, professional development, and 
curriculum integration. 
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In this section, we will use three intermediate goals and two outcome goals as 
examples: 



Intermediate 

Professional 

Development 


All teachers will have the training they need to help students 
learn using computers and the Internet. 


Availability of 
Technology 


All teachers and students will have modern multimedia 1 

computers in their classrooms. J 


Curriculum 

Integration 


Teachers will use and view technology (e.g., effective and 
engaging software and online resources) as an integral part of 
the curriculum and daily classroom activities. 


Outcome 


Reading and 
Writing 


Student reading and writing will significantly improve. 


Computer 

Literacy 


Both teachers and students will display increased computer 
literacy. 



Although there are many different ways that the above goals could be related, 
the following diagram outlines some of the possibilities: 






Outcome Goals 










Reading and 
Writing. (Student 
reading and writing 
will significantly 
improve.) 





Computer Literacy. 

(Both teachers and 
students will display 
increased computer 
literacy.) 



The relationship among the goals is not straightforward. For example, all of 
the intermediate goals can work together directly to influence the computer 
literacy of students, and both professional development and the availability of 
technology can influence computer literacy by means of their effect on 
curriculum integration. Every goal (intermediate and outcome) is important, 
and it is necessary to assess each of them. We now turn to the task of how 
to assess goals and, specifically, the task of developing indicators, 
benchmarks, and measures based on goals. 
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I] Next, you will need to develop indicators. For our purposes here, 

INDICATORS 1 indicators are statements that reflect specific goals that can be used to gauge 

progress. Indicators help orient you toward a measure of performance 
outcomes and typically focus on only one aspect of a goal. Some examples 
related to the goals described on the previous page are: 



Goals Indicators 

Intermediate 


Professional 

Development 


• Increasing numbers of teachers will be trained to integrate 
technology into their teaching. 

• Teachers will learn how to integrate technology into their 
teaching. 


Availability of 
Technology 


• Teacher:computer and student:computer ratios will 
steadily decline. 

• Students will actively use computers for projects and 
assignments. 


Curriculum 

Integration 


• Technology will increasingly be incorporated into the 
curriculum in all subject areas. 

• An increasing percentage of teacher lessons will 
incorporate technology. 


Outcome 


Reading and 
Writing 


• Students will produce reports and presentations that 
teachers judge to be of higher quality, 
e Students will display increasingly higher performance on 
tests assessing reading ability. 


Computer 

Literacy 


• Both teachers and students will display increased 1 

computer literacy. 1 



Good indicators should: 

• Include relevant measures. Indicators should be designed with thought 
toward the information that could be collected to inform them. 

• Address appropriate groups. Indicators should consider any 
subpopulations that need to be examined separately (e.g., school level, 
racial/ethnic group). 

• Be few. You do not need a lot of indicators to convey a lot of information. 

• Be actionable. Indicators should guide improvement by helping to identify 
programs that are working and those that are not, should help program 
administrators make decisions to improve results, and should help policy 
makers judge progress toward goals. 

• Be timely. Indicators should be as useful next year as they are today. 

• Be reliable. Indicators should be easily measured by a variety of people 
and across time. 

• Be comparable. Measures developed from the present indicators should 
be able to be compared to measures developed from similar indicators in 
the past. 
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BENCHMARKS a benchmark is a specific target. Benchmarks provide a sense of what the 

program is striving to attain, either on an incremental basis (e.g., “every year 
test scores will rise 2 percentage points”) or as a final target (e.g., “all 
students will score at or above the proficient level”). Two important factors 
need to be considered when setting benchmarks: the resources available to 
the program and a starting point, or baseline. If, for example, you are 
evaluating a relatively small initiative that is funding the purchase of new 
equipment, you would not expect the student to computer ratio to increase to 
the same extent as it might in a program that provides more extensive 
funding. Baselines also are important. They help establish the amount of 
change one can realistically expect. Let’s assume that the purpose of the 
program you are evaluating is to improve student reading. To set an 
appropriate benchmark, you would need to determine student performance 
before the program began. This is considered a “baseline,” a point from 
which to measure change and to determine appropriate benchmarks. 



Indicators Benchmarks I 

J Intermediate 1 


o increasing numoers or teacners — 
will be trained to integrate 
technology into their teaching. 


■"* Byrne end of the school year, 50 
percent of teachers will have had 
technology training. 




• Teachers will learn how to 
integrate technology into their 
teaching. 


• After training, teachers will be 
able to devise at least three ex- 
amples of how technology could 
be integrated into their lessons. 




• Teacher:computer and student: 
computer ratios will steadily 
decline. 


• By next year, teacher:computer 
and student:computer ratios 
schoolwide (counting all 
computers) will be 4:1 and 6:1. 


• Students will actively use 
computers for projects and 
assignments. 


• All students will use computers at 
school at least 4 hours per week. 


• Technology will increasingly be 
incorporated into the curriculum in 
all subject areas. 


• In two years, English, science, 
and social studies curricula will 
have at least 25 percent of 
lessons incorporating technology. 


» An increasing percentage of 
teacher lessons will incorporate 
technology. 


• All trained teachers will have at 
least 25 percent of lessons 
incorporating technology. 


routeome \ $«• ’ 


° Students will produce reports and 
presentations that teachers judge 
to be of higher quality. 


• Within two years, student grades 
on reports and presentations will 
increase, on average, by 10 
points or one letter grade. 1 


• Students will display increasingly 
higher performance on tests 
assessing reading ability. 


• Within two years, student scores 
on the reading portion of 
standardized tests (e.g., ITBS, 
CAT) will increase by 10 percent. 1 


• Both teachers and students will 
display increased computer 
literacy. 


• After one year, at least 75 percent 
of teachers and students will 
display at least an intermediate 
level of computer literacy. 



erIc * 20 



1 Figures are used as examples only. The actual percentage difference that would be considered 
significant will vary by a number of factors (e.g., number and type of students tested, test or grades used) 
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A measure is an item reflecting the information or evidence needed to 
answer a research question , inform an indicator, or determine how close 
one is to achieving a benchmark. If applicable, a measure typically 
includes gauges such as percentages and ratios. Measures are similar to 
indicators, but are more concrete and more specific. Often, an indicator will 
have multiple measures. Measures attempt to answer the question of what 
evidence you would need to have to assess your indicators. The following are 
examples of measures used to assess the benchmarks: 



Benchmarks | Measures 


Intermediate 


« By the end of the school year, 50 j • Number and percent of teachers 

percent of teachers will have had j requesting and receiving training 

technology training. i by type of training 


• After training, teachers will be 
able to devise at least three 
examples of how technology 
could be integrated into their 
^ lessons. 


• Lesson plans produced at 
professional development 
activities 

• Teacher responses from 
professional development survey 


• By next year, teacher:computer 
and student:computer ratios 
schoolwide (counting all 
computers) will be 4:1 and 6:1. 


• Teachercomputer ratios 

• Student:computer ratios 


• All students will use computers at 
school at least 4 hours per week. 


* Percent of students who use 
computers at least four hours per 
week at school 


• In two years, English, science, 
and social studies curricula will 
have at least 25 percent of 
lessons incorporating technology. 


• By subject area, the percentage 
and frequency of lessons included 
in the curriculum that incorporate 
technology i 


• All trained teachers will have at 
least 25 percent of lessons 
incorporating technology. 

I 


• From classroom observations, the 
percentage of teacher lessons 
that incorporate technology 

• Percentage of lesson plans that 
incorporate technology 


Outcome 


• Within two years, student grades 
on reports and presentations will 
increase, on average, by 10 
points or one letter grade. 


• Student letter or numerical grades 
on reports and presentations 


• Within two years, student scores 
on the reading portion of 
standardized tests (e.g., ITBS, 
CAT) will increase by 10 percent. 


• Student standardized test scores 


• After one year, at least 75 percent 
of teachers and students will 
display at least an intermediate 
level of computer literacy. 


• Student and teacher self-report of 
computer literacy 

• Student and teacher results from 
a skills test requiring performance 
of various tasks on a computer. 



On the next page, you will find a summary of the goals, indicators, 
benchmarks, and measures discussed. This should help you to follow your 
own path from general goals to specific ideas of what information you will 
need to collect to measure them. Take it one step at a time. First, write down 
your goals and questions, then indicators, benchmarks, and measures. 
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Goals© Indicators 


.Benchmarks 


1 Measures ! 


Intermediate 


— 


Professional 

Development 


• Increasing numbers of 
teachers will be trained to 
integrate technology into 
their teaching. 


° By the end ot the school 
year, 50 percent of 
teachers will have had 
technology training. 


• Numoer ana percent ot 
teachers requesting and 
receiving training by type of 
training 


• Teachers will learn how to 
integrate technology into 
their teaching. 


• After training, teachers 
will be able to devise at 
least three examples of 
how technology could be 
integrated into their 
lessons. 


• Lesson plans produced at 
professional development 
activities 

• Teacher responses from 
professional development 
survey 


Availability of 
Technology 


• Teacher:computer and 
student: computer ratios 
will steadily decline. 


• By next year, teacher: 
computer and student: 
computer ratios school- 
wide (counting all 
computers) will be 4:1 
and 6:1. 


• Teachercomputer ratios 

• Student:computer ratios 


• Students will actively use 
computers for projects 
and assignments. 


• All students will use 
computers at school at 
least 4 hours per week. 


• Percent of students who 
use computers at least four 
hours per week at school 


Curriculum 

Integration 


• Technology will 
increasingly be 
incorporated into the 
curriculum in all subject 
areas. 


® In two years, English, 
science, and social 
studies curricula will have 
at least 25 percent of 
lessons incorporating 
technology. 


• By subject area, the , 
percentage and frequency 
of lessons included in the 
curriculum that incorporate 
technology 


• An increasing percentage 
of teacher lessons will 
incorporate technology. 


® All trained teachers will 
have at least 25 percent 
of lessons incorporating 
technology. 


• From classroom 
observations, the 
percentage of teacher 
lessons that incorporate 
technology 

• Percentage of lesson plans 
that incorporate technology 


Outcome ■ ..j* i . 


Reading and 
Writing 


® Students will produce 
reports and presentations 
that teachers judge to be 
of higher quality. 


• Within two years, student 
grades on reports and 
presentations will 
increase, on average, by 
10 points or one letter 
grade. 


• Student letter or numerical 
grades on reports and 
presentations 


• Students will display 
increasingly higher 
performance on tests 
assessing reading ability. 


• Within two years, student 
scores on the reading 
portion of standardized 
tests (e.g., ITBS, CAT) 
will increase by 10 
percent. 


• Student standardized test 
scores 


Computer 

Literacy 


• Both teachers and 
students will display 
increased computer 
literacy. 


• After one year, at least 75 
percent of teachers and 
students will display at 
least an intermediate 
level of computer 
literacy. 


• Student and teacher self- 
report of computer literacy 

• Student and teacher results 
from a skills test requiring 
performance of various 
tasks on a computer. 
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What Information Did Rivers Need to Collect? 

Kathy decided to take the goals and questions she had already come up with and use the goals to 
develop indicators, benchmarks, and measures. Here is what she came up with: 





Indicatory 


Benchmark*' 


Measure*' 


Improved/ reading 
performance/ (Will/ 
the/ new technology 
andtraining 
improve/ readings 
performance/?) 


Test scores in 
readingwill 
improve/ for all/ 
Students, 
including those/ 
who- display poor 
performance/. 


In/ 3 years, scores 
will/ improve/, on/ 
average/, by 10 
percent with/ at 
least a/10 
percent increase/ 
also- seen in 
students scoring 
in/the/lowest 
quartile/ (bottom/ 
25 percent) 


• ITBS scores in 
reading 

• State/test scores 
in reading 

• Grades in 
reading 


Improved/ dropout 
and/ attendance/ 
rate y ( Will/the > new 
technology and/ 
traininghelp lower 
the/ dropout rate/ 
and/ increase/ 
attendance/?) 


Percentage/ of 
dropouts will/ 
decrease/ and/ 
Attendance/ will/ 
increase/. 


By the/ end of the/ 
school f year, the/ 
dropout rate/ will/ 
decrease/ 1 
percent and the/ 
attendance/ rate/ 
will/ increase/ 2 
percent. 


• Dropout rate/ 

• Attendance/ 
rate/ 


Cost-effectiveness 
(Willthisprogram/ 
be/asor more/ cost- 
effective/ than other 
programs that may 
show similar 
results?) 


The/program/ will/ 
be/ as or more/ cost’ 
effective/ than 
other programs 
withsimilar 
results. 


The/budget for 
this program/ will/ 
be/ the/ same/ as or 
lower than a/ 
program/ 
demonstrating 
Similar results. 


• Program/ 
budget 

• Budget of slmr 
dar program/ 

• Program/ 
evaluations 


Effective/professionat 
development 
activities (Will the/ 
professional 
development help 
teacher y integrate/ 
technology inter 
their teaching?) 


Teachers will/ 
develop lessons 
that integrate/ 
technology into 
their teaching. 


Teachers will/ 
develop lessons 
that integrate/ 
technology into 
their assign- 
ments at least 
once/ a/ week/. 


• Review of 
teacher lesson 
plans 

• Observation of 
classrooms 

• Teacher Survey 


Increased/ computer 
literacy (Will/ 
Studentyand 
teacherybecome/ 
more/ computer 
literate/ as a/ result of 
the/program/? ) 


Teachers and 
studentywill/ 
become/ more/ 
computer Uterate/. 


By the/ end of the/ 

school/year, 

teachers and 

studentswill/ 

show improved 

computer 

literacy. 


• Assessment of 
teacher 
computer 
literacy 

• Assessment of 
Student 
computer 
literacy 
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"What Information Do S Meed to Collect?" Worksheet 

Using the goals and research questions you came up with in the last section as a starting point, 
work through the following grid. You may have to revise some of your goals and questions as a 
result. 



Goals 


Indicators 


Benchmarks 


Measures | 
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Baseline Data \ 



Pilot Testing 



Using 

Multiple 

Sources 



What’s the Best Way to Collect My Information? 



There are many different methods of collecting information. Any college 
textbook on research design and analysis will provide an overview of different 
methods of data collection (e.g., Rossi & Freeman, 1993). Here, we present 
the "highlights." We will begin with a few tips, or words of advice, and in the 
process define some basic terminology. 

Collecting baseline data may be important. As noted in the discussion on 
benchmarks on page 20, baseline data provide information at the start of a 
program. Baseline data are used to set benchmarks and to determine the 
amount of change at different points .in the course of the program. For 
example, if you wanted to find out if teacher use of technology increases after 
professional development, it would be best to collect information on use both 
before and after the activity. In this case, information on the amount of use 
before professional development would be considered "baseline data." 
Baseline data are most important when the question you are trying to answer 
is one of change (e.g., has student computer use increased? Does training 
help improve teacher attitudes towards technology?). 

Pilot testing may prevent costly mistakes. Another term that you may 
have heard is "pilot testing." Pilot testing is a trial run of procedures and 
instruments that you plan to use. For example, pilot testing of a mail survey 
to teachers could be done in a couple of different ways. You could mail the 
survey to a handful of teachers and then call them to discuss the questions. 
An easier, but less rigorous, method might be to distribute the survey after a 
professional development activity and ask the teachers there what they think. 

The main purpose of pilot testing is to catch potential problems before they 
become costly mistakes. It is typically used if an instrument or method of 
data collection is being used for the first time or for the first time with a 
particular group (e.g., a survey used before with a different age group). Pilot 
testing provides information on how long data collection can be expected to 
take and a preview of how difficult items will be to complete. The latter is 
important as, with proper advanced notice, you can modify questions and 
possibly even the way you collect information (e.g., reading questions to 
people rather than having people read questions themselves). 

It’s best to use multiple sources. When looking for evidence, many 
sources are better than one. For example, if you are gathering information on 
student computer use and you obtain reports from students, observe students 
using the computer, and get computer logs detailing an increase in use, there 
is a good chance that the information is reliable. On the other hand, if only 
computer logs are collected, or only student reports or observations are used, 
you will need to make conclusions more cautiously. A general rule is, the 
more sources of evidence that you have, the more faith you can have in a 
result or finding. 
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Sampunq ; ,,’j Sometimes you won’t have the time or money to get information from 

everyone. If collecting information from everyone (e.g., all schools) is not 
feasible, you will need to sample or get information from a subset of persons 
or schools. Proper sampling techniques allow you to take information from a 
small group and generalize to a larger group (e.g., some students in a school 
to all students in the school). 

There are many ways to sample, and the process can readily become difficult. 
By far, the easiest way to is to choose people randomly (e.g., pull names out 
of a hat or take every third person on a list). However, sometimes random 
sampling may not be possible (e.g., you need to rely on volunteers, groups 
may already have been assigned), or there may be a reason that you would 
not want to sample randomly. For example: 

• You may be interested in results from a particular group. For example, you 
may be particularly interested in learning whether your technology 
program improves student achievement for students who transferred to 
your high school from a particular middle school, but there are not that 
many students who participated in your technology program from that 
school. If you selected a random sample, you may not select enough of 
these students from which to draw conclusions. To investigate the effects 
of your technology program on students from that particular middle 
school, you may need to sample these students separately or only look at 
these students. 

® You may want to make certain that your sample reflects a range of 
students or teachers. For example, you may want to find out if a 
technology program works with students at all ability levels. A true 
random sample from the district as a whole may not provide you with 
sufficient numbers of low-performing students or students enrolled in 
honors classes. You thus may need to “oversample” low-performing 
students or students enrolled in honors classes. 

Techniques of sampling that would not be considered random are used fairly 
regularly, but must be treated with caution. By not selecting a sample 
randomly, you will be able to make conclusions about the types of students or 
teachers that were sampled, but you may not be able to make conclusions 
about the group as a whole (e.g., all students in the district). 

Sampling can be a valuable tool and there are many resources available to 
help you (e.g., books, statisticians or social scientists at your local university). 

A good piece of advice on sampling is to consult your library or call a local 
university if you have questions. 
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i Groups 



You may need to compare one group to another. Often, it is helpful when 
evaluating a program to compare one group to another. Typically, 
comparisons are made between students or teachers who have been 
exposed to a particular initiative and students or teachers who have not been 
exposed. For example, you might compare: 

• Classroom lessons of teachers who have received training to lessons of 
teachers who have not received training: 

• The computer literacy of teachers before and after professional 
development: 

• The computer literacy of students in classrooms with four computers to 
students in classrooms with only one computer; of 

Other types of comparisons can also inform evaluations. You might want to 
compare: 

• The amount and type of computer use of girls to that of boys; or 

• The achievement of students in the school to national norms. 

As you can see, there are many ways to think about comparison groups. 
Some require looking at possible differences in the same group over time 
(e.g., comparing teachers before and after a professional development 
activity). Some require examining differences between groups differing in 
program implementation (e.g., number of computers in a classroom). Some 
require looking at how different groups (e.g., gender, ethnicity, at-risk, grade 
level) respond to the same program. All methods are used regularly, and the 
“best” method will vary from program to program. In general, there are three 
pieces of advice we would give: 

• Compare groups that are as similar as possible in areas other than the 
difference you are investigating (e.g., someone interested in gender 
differences in computer use would make sure that students differed only 
on gender). 

• Make conclusions cautiously about the role technology plays in 
differences (e.g., What else could explain differences?). 

• Compare your results to those of other studies (e.g., Did other people find 
the same thing? Why or why not?). 
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Source of ; j 

Information j 



, Comparing to 
Other 
Results 



j Possible 
Constraints 



Designing for 
Analysis 



To figure out which method of data collection to use, you will need to ask 
yourself some additional questions. Specifically, you will need to ask whom 
you will get your information from, whether you will need to compare your 
program with others, and what possible constraints there are. 

Whom should I be getting information from? Your source of information 
will vary depending on the issue you wish to address. For example, if you are 
interested in professional development, it would be best to get information 
from teachers and/or administrators or coordinators. If you are interested in 
student use, then it would be wise to go to the source - students. For 
technology programs, possible sources of data include administrators, 
teachers, students, parents, program coordinators, businesspeople (e.g., 
software manufacturers), and financial experts (e.g., information from 
budgets). There are three main things to keep in mind when deciding what 
source of information to use. First, think about the person(s) or materials that 
will give you the most accurate information. Second, assess the availability of 
this person or materials and use an alternate source if necessary, staying as 
close to your desired source as possible. Third, tailor your instruments to the 
level of knowledge of the person you are getting your information from. For 
instance, you will need to word a questionnaire very differently for adults than 
for children. 

Do I need to be able to compare my results to others? If you need to be 
able to compare across programs or compare the results of your evaluation 
with similar evaluations, you should look at how information was collected 
previously and what instruments were used. If the same method and 
instruments are available and will work, use them! Otherwise, you will need to 
stick as closely as possible to what was done previously and make 
comparisons cautiously. 

What are some possible constraints? Many of the reasons that 
researchers choose one data collection method over another have to do with 
different constraints that were placed on them. Some typical constraints 
include time, money, availability of personnel, and legal or bureaucratic 
roadblocks (e.g., not able to use a copyrighted test, needing permission from 
parents and/or school officials but there is not time). Two common con- 
straints are the need for district approval (e.g., the district must approve any 
research done in the schools) and parental permission for student surveys 
(e.g., do you need signed permission slips?). Check out these policies early! 

What kind of analysis will the information require? When deciding what 
type of information to collect, another point to consider is the type of analysis 
that will be required to form conclusions about the information. For informa- 
tion such as numerical responses on surveys, you may want to think about 
the level of statistical knowledge required. If your background is not exten- 
sive, it may be wise to gather information that can be expressed in terms of 
counts or percentages (e.g., 46 percent of teachers said they had training). 

For qualitative types of information, time and complexity of analysis may both 
be a concern. Coming up with a method of looking at and coding narrative 
responses can be both a time-consuming and complicated task. On the one 
hand, qualitative information is useful as it provides a lot of detail and rich 
context. However, analyzing it is often not as straightforward as it seems. 
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Now that you have your potential measures and an overview of some basic 
data collection issues, it's time to look at your options for what method to use 
to collect information. Some common methods include surveys, interviews, 
focus groups, observations, and collection of already available materials (e.g., 
school records). Following is an overview of these different methods of data 
collection. 

SURVEYS I Surveys are often used to collect information from large groups. 

Surveys can vary along a number of dimensions, such as how questions are 
asked and how the survey instrument is delivered. For instance, questions 
can be open-ended (no choices provided) or close-ended (choices provided). 
Surveys can also be mailed or distributed in teachers’ boxes at school, or 
administered by telephone, or in a large group setting. 

The question of what to ask on a survey and how to ask it is not an easy one. 
Like sampling, entire books and college courses have been devoted to the 
development and delivery of surveys. It is easy to get lost in the complexity, 
but there is a basic tenet that may help you out:. Think about what you need 
to know and ask it as directly as you can. For example, if you want to know 
how often teachers use the Internet in their teaching, you may want to ask, 
“How often do you use the Internet in your teaching?” either in an open-ended 
format or by providing possible response options (daily, once a week, etc.). 

Once you have survey questions that you feel will meet your needs, you will 
need to figure out the best method to administer them. The three most 
common ways to deliver a survey are by mail or distribution into "mailboxes" 
at a school, in a group setting, or by telephone. Mail surveys are one of the 
least costly methods of delivering surveys. However, you may run into 
problems getting people to return surveys, obtaining correct addresses and 
phone numbers. Administering a survey to a group, possibly at a teachers’ 
meeting, is probably the quickest way to collect survey information. In 
addition, administering a survey to a group will allow more flexibility than mail 
surveys to answer respondents’ questions (e.g., "Do I consider graphing 
calculators as a use of technology?"). Telephone surveys may be a good way 
to go if the group of people you wish to collect information from is not readily 
available (e.g., parents) or if the information requires extended explanation on 
either your part or the part of the respondent. 

Overall, there are many advantages to using surveys. Most are easy to 
analyze, they can be quickly administered, and they can be easily adapted to 
fit specific situations. Using a previously administered survey will save you 
time and energy, may give you something to compare your results to (if 
results from previous administrations of the survey are available), and will give 
you confidence that some of the potential problems have already been 
addressed. One word of caution in using surveys that others have developed 
is warranted, though. It’s tempting to look at an already developed survey 
and think that it is wonderful, but you need to look at what information you 
need and adapt surveys as you see fit. 
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Just like surveys, interviews can vary greatly in form. Interview protocols 
can range from extremely unstructured (e.g., a list of general topics to be 
covered) to something that might resemble reading aloud a survey (i.e., lists 
of questions that must be asked). Interviews are somewhat like surveys - an 
interviewer asks questions of people - but since interviews occur on an 
individual basis, they are often more time-consuming and costly. In addition, 
evaluators need to be concerned about consistency across interviewers. 
However, interviews also have advantages. They are good for collecting 
detailed information or information from a limited number of people. In 
addition, interviews give people an opportunity to explain responses or give 
responses that you may not have predicted. For example, interviews may be 
useful in examining gender differences in signing up for computer classes. 

Focus groups can be an easier way to collect information that otherwise 
would be collected through interviews. Focus groups can be thought of as 
group interviews. Just like interviews, they can be structured or unstructured. 
One of the most popular ways to use focus groups is in conjunction with 
surveys - either the focus group helps the researcher interpret the results of a 
survey or a focus group helps define and/or refine survey questions. An 
advantage of focus groups is that, since data are being collected from 
multiple people at the same time, they are more cost-effective than interviews. 
A disadvantage is that, in the group setting, you might not get to hear the 
opinions of all participants (e.g., some may not feel comfortable speaking in 
this forum). You also need to be concerned about how well those 
participating in the group represent the group you are interested in (e.g., are 
there only “honors” students in the group?) Another issue regarding focus 
groups, that can be both an advantage and a disadvantage, is that responses 
from one participant can sometimes trigger responses from others. 

Another common method of collecting information is to observe. 

Sometimes the best way to collect information about people's behavior is to 
watch them. Observation allows the researcher to collect information without 
being a burden on the person providing the information. Typically evaluators 
develop guides that structure the observation process. As an example, 
observation can be used to collect data on teacher lessons and student use of 
computers. One drawback to observation, however, is that it measures only 
what you can see. Other types of data (e.g., opinions, reasons behind 
behavior) cannot be collected in this fashion. Another drawback is that it can 
be time-consuming as multiple observations are often required. 

Don’t forget school records. Schools routinely gather lots of information 
(e.g., attendance, grades, test scores) that could be useful to your evaluation. 
Getting this information directly from school records can save time. In 
addition, it can also be more accurate than asking students or teachers for the 
information and relying on their recollections. However, it is important to 
ensure the confidentiality of school records. In general, you will need to make 
sure that you don’t report individual student names, that you have accessed 
records through the appropriate channels, and that you have received 
necessary permission. Even with these caveats in mind, though, records may 
be easier to get than you may think and, if they provide needed information, 
they are worth considering. 
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Finally, you could also look at other existing documentation. Looking at 
already existing evidence (e.g., teacher lesson plans) places little burden on 
people. Drawbacks, however, are that information may not be exactly what 
you need, in the form you need, or may be incomplete or need explanation. 

Different pieces of evidence take different amounts of time to collect. 
Often stakeholders will want to see results quickly. For some measures (e.g., 
test scores), however, it could take years. What do you do in the meantime? 
Here are some ideas for evidence that doesn’t take long to see results: 

• Information on student use. For example, gather narratives on time spent 
at computers, how students are using technology (e.g., class projects), 
anecdotes of student excitement (e.g., “they are lined up during lunch 
period”) or student problems (e.g., getting into inappropriate material). 

• Student work. Soon after installing equipment, students will begin to use 
it to produce work. Collecting this work will provide examples of how the 
equipment can be used and the possibilities for improving student work. 

• Teacher lesson plans or classroom observations. Soon after teachers 
have been trained, some will experiment with technology in their lessons. 
These beginning lessons can be .used as either examples for further 
training or examples to give to stakeholders of technology use. 

• Comments from students and parents. Like it or not, teachers and others 
will hear, comments (either positive or negative) from students or parents. 
Ask around to find out what these comments are. Comments might give 
you ideas on areas you may want to investigate later or they may provide 
anecdotal evidence of student use and attitudes. 

• Start collecting outcome data. Although it may be too soon to answer 
impact questions, you can start collecting information relating to outcomes 
early in the program’s history. Gathering information from the beginning 
of the program will allow you to track students’ progress over time. 

Collecting a variety of information throughout a program forms the core of an 
evaluation approach known as “portfolio assessment.” For more information 
on portfolio assessment, consult the RAND website (http:/www.rand.org). 

At this point, you have information on many possibilities. The next step is to 
put the information together to decide which of the many possibilities to use. 
On the following page is a summary of the different methods of collecting data 
that were discussed here, listing some advantages and disadvantages for 
each. It is not meant to be an exhaustive list, but rather a guide for things you 
should watch for or be aware of. 

In addition, you will find, in Appendix C, examples of surveys. If they address 
the information you are looking for, great! Acknowledge where they came 
from and feel free to use them. For other materials, you need to make certain 
that there are no copyright restrictions. 

After some final decisions, you should be ready to go. You have your goals 
and research questions, you know what kind of information you need, and you 
know how you're going to collect it. Remember to make sure you have all the 
permissions necessary (e.g., district, teachers, parents, children), and start 
interviewing, surveying, observing, or collecting materials. 
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Different Methods of Collecting Information 


Method 


Advantages 


Disadvantages 


Surveys Mail 

Group- 

Administered 


® Can survey many people 

• Not time-consuming 

• Relatively inexpensive 

• Everyone gets the same instrument 

• Objective Interpretation 


• Difficult to get much detail 

• Sometimes difficult to get correct 
addresses 

• May be problems with interpreting 
questions 

• Sometimes a problem getting 
surveys completed and returned 


• Can survey many people 

• Not time-consuming 

• Relatively inexpensive 

• Everyone gets the same instrument 

• Objective interpretation 

• Relatively inexpensive 


• May be problems with interpreting 
questions 

• Difficult to get much detail 


Telephone 


• Able to ask for more detail when 
needed 

• Everyone gets the same instrument 


• Sometimes difficult reaching people 

• Lack of anonymity 


Interview 


• Researcher can know how people 
are interpreting questions 

• Able to ask for more detail when 
needed 

• Provide detailed data 


• Time-consuming 

• Because of time, can limit sample 
size 

• Subjective interpretation 

• Can be expensive 

• Can be difficult to analyze 


Focus Group 


• Researcher can know how people 
are interpreting questions 

• Able to ask for more detail when 
needed 

• Able to interview multiple people at 
one time, thus, more cost-effective 

• Responses from one person provide 
stimulus for other people 


• Group setting may inhibit some 
individuals from providing 
information 

• Sometimes hard to coordinate 
multiple schedules 

• Responses from one person provide 
stimulus for other people 


Observation 


• Objective interpretation 

• Low burden for people providing 
data 


• Time-consuming 

• Some items are not observable 

• Can be expensive 

• Participant behavior may be 
affected by observer presence 


Student Records 


• Objective interpretation 

• Low burden for people providing 
data 

• Relatively inexpensive 


• May not correspond to exactly what 
researcher wants 

• May be incomplete or' require 
additional interpretation 

• May need special permission to use 


Collection of Materials 


• Objective interpretation 

• Low burden for people providing 
data 

• Relatively inexpensive 


• May not correspond to exactly what 
researcher wants 

• May be incomplete or require 
additional interpretation 
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How Did Rivers Collect Information? 



Kathy listed possible ways to collect information pertinent to her measures and came up with the 
following: 

• Review erf teacher th^ toobpart Cvu profeyyiOYial 

develop m acXZvUXefr 

• IwtervCewy - pewevrfy, tectchery, ytudevxty, admiAxi^atory 

• Collect: reculCng^ teyt bcorey 

• Collect lAxforvvuvUoyv from/ dlytrict recordy on/ cUtendavice cund/ dropout raley 

• Aaeafnent of computer literacy - Observation/? Self-Assessmert? Teyt? 

Since her principal said that she had only had six months to come up with a report, she decided 
that she could not do everything she wanted - doing both interviews and surveys would take too 
much time. So, she had to set priorities. Well, since surveys are quicker, she decided to survey a 
random sample of 100 teachers, 25 administrators, 300 students, and the parents of the 300 
students. While searching the Internet, Kathy found examples of surveys (in fact, many more than 
she needed!) that had previously been used both in high schools and in elementary schools. She 
found a student survey that looked like it would work well - it had some questions on attitudes 
toward computers, use of computers, and a self-assessment of computer literacy. Her district had 
assessed professional development programs before, so a survey was available that had been 
previously used. But in order to address her specific professional development concerns (e.g., will 
professional development help teachers integrate technology into their teaching?), Kathy needed 
information on both participation and its results. She therefore needed to review teacher lesson 
plans and observe teachers in the classroom as well as determine how much and what types of 
professional development teachers participated in. 

Kathy felt she came up with the right set of instruments and methods to use, and since most of the 
survey questions had been used before, she felt pilot testing was not necessary. However, she 
still had a few questions: 

• WhathOnd/ of parental and/ or dtytrictpermlMtOYv'would/be needed/? 

• Arvu I ^oin^ ter ran/ Crto-ytatUtU^d' analyses I carlt handle? 

• Will/my re^ultybecervnparable to- other ytudley? (people \v art thiy) 

Kathy, luckily, has a friend at a local university. She faxed drafts of her surveys to him and asked 
his opinion on comparability, statistical analysis, and permissions needed. He assured her that, 
since the issues addressed were not sensitive (e.g., sex, drugs) and since she worked for the 
district, she did not have to go through a formal review process to get district approval. In addition, 
she could handle parent permission by sending the parent surveys out first and giving them a 
vehicle for declining to let their children participate (e.g., call her if they have a problem with it). 

Finally, Kathy's friend at the district said that she did the right thing by picking previously used 
surveys, and that since the district had used them before, there would be no problem with 
copyright. He suggested a few changes to the questions she had added, but basically she was 
"good to go" and would be able to compare her results to other programs. 
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“What’s the Best Wayto CollectMy Information? ^Worksheet 

What did you think of as possible measures? 



What kind of information has been collected before and/or is already available? 



What kind of constraints do you have (e.g., time, money, needed permission from district)? 



Assess Quality of Measures. Will the measures give you the information needed? Are they 
doable given time and financial constraints? 



Do you need to sample? If so, how will you do it? 



Do you need to use comparison groups? If so, how will you do it? 



Have the instruments been used before? If not, do you need to do pilot testing? 



What kind of statistical knowledge will be required to analyze the data? 



Any other problems you need to resolve? 
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What Are My Co nclus ions?! 



Review n 
Research | 

Questions i 

.£ 



What’s next? I've got some information, now what do I do with it? The first 
thing you should probably do is review your goals and research questions. 

Figure out what you wanted to know in the first place. It's easy to get lost in the 
amount of data you have collected and the details of individual responses. 
Reviewing what you wanted to know may help focus you on the task at hand - 
figuring it out! 



Organize 

Your 

INFORMATION 



Next, organize your information by goal. If you filled out a chart of goals, 
indicators, benchmarks, and measures, you have a good start. Make a list of 
what information you now have that corresponds to each measure, and thus, 
addresses each of your goals. You will probably find that one instrument, such 
as a survey, addresses multiple goals and helps to answer multiple questions. 
This is expected. Do the best you can to separate out which measure 
corresponds to which goal and if there is overlap (e.g., one measure still 
addresses multiple goals), put the measure under both. 



Like it or not, the next step is data analysis. Sometimes, when people think 
ANALYZE Of data analysis, they assume that complicated statistics must be involved and 

are worried that they do not have the background to attempt such a task. In 
reality, there are two things to keep in mind: 



• Not all data analysis involves complex statistics. 

• Even if statistics are involved, they should be at the level that the intended 
audience will easily understand. 



The most common way statistics are used in evaluation is for descriptive 
purposes. For example, if you want to describe the number of hours students 
spent using a computer at school, you would want to calculate either an average 
or the percentage of students who use computers more than a certain specified 
base. In addition, you may want to also compare the results of different groups 
of students (e.g., at-risk students, gifted and talented students) to see if 
technology has different impacts. In this case, you may want to use the same 
statistics (e.g., means, percentages), but report separate results for each group. 
Whether you use an average or percentage and how you want to group your 
information depends on the type of information you have collected. 



. Coming to a 
I Conclusion 



In the end, the important part of analyzing information is not the statistics, but 
rather the conclusions that you draw. This is what this section is meant to guide 
you through - reaching conclusions. 



The process of coming to a conclusion can vary widely from goal to goal. One of 
the most difficult tasks is defining vague goals, such as what is "sufficient 
training"? Luckily, you have your indicators and benchmarks to help you with 
this. For example, if a certain benchmark was reached, then what you were 
measuring could be considered “sufficient” or “adequate.” 



To help you understand the process and give you some ideas, several examples 
of how goals were addressed are provided on the next page. The goals vary in 
topic and type of information used. 
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Goal 


Measure(s) j Coming to a Conclusion 


Professional 

Development 


• Number and 
percentage of 
teachers • 
requesting and 
receiving training 
by type of training 


Carlos in District A wanted to know about teacher access to 
training. The district wanted to make sure that all teachers had 
training within two years, so he developed a benchmark - 50 
percent of teachers should have had training after the first year 
- to determine the extent of training. Carlos then obtained a 
list of all training activities offered at the state, district, and 
school levels from the district professional development 
coordinator and lists of persons who attended the activities. 

He found out that there was a three-day summer training given 
by the district (50 slots open) and three one-day workshops 
given by one of the schools in the district (30 spaces for each). 
The three-day summer training ended up being full (50 people 
attended), but only an average of 20 people attended the 
workshops. Carlos was then able to describe what was offered 
and the number of attendees and compare what he found with 
the indicators and benchmarks that he had established. With 
only 25 percent of teachers having had training, he concluded 
that, unfortunately, the district was not meeting its benchmark. 
He recommended that, since the summer workshop was well 
attended, the district should offer additional summer 
workshops in the coming year. 


Curriculum 

Integration 


• By subject area, 
the percentage 
and frequency of 
lessons included 
in the curriculum 
that incorporate 
technology 


Cecily in District B sat down and went through the district 
curriculum. Defining “incorporating technology” as either the 
teacher or students using technology during the proposed 
lesson or activity, and having set a benchmark of 25 percent of 
lessons as optimal, she counted all the lessons in three main 
subject areas: English, science, and social studies. She found 
that 30 percent of the science and social studies lessons 
contained technology, but only 5 percent of the English lessons 
contained technology. Thus, Cecily concluded that science 
and social studies are on target, but English is not. 


Reading and 
Writing 


• Student letter or 
numerical grades 
on reports and 
presentations 


Steve in School C asked all students in the fourth and fifth 
grades to write three reports at the beginning of the school year 
on different trips they would like to take within the United 
States. At the end of the school year, he asked these same 
students to write three reports on trips they would like to take to 
other countries. He not only examined these reports for 
content and quality of citations, but also incorporated 
information into his conclusions on the extent to which the 
teachers of these students used technology in their 
classrooms. (He had observed each of the fourth and fifth 
grade classrooms on at least six random occasions throughout 
the year.) Students whose teachers incorporated technology 
into their lessons did improve more than students whose 
teachers did not incorporate technology as much. Thus, Steve 
concluded that having computers available to students in 
classrooms was not sufficient; teachers had to use technology 
on a regular basis in their lessons in order for student 
performance to improve. 
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What Were Rivers’ Conclusions? 

Here is Kathy's attempt to come to conclusions on her research questions: 



Good/ 


Meaaw&Oi) 


Conclu4uyy\/Proce4i' j 


Improved/ readings 
performance/ (Will 
the/ new 

technology and/ 
trainings improve/ 
readings 
performance/?) 


• ITBS readings 
scores' 

• State/ test scores' in 
readings 

• Grades' in readings 


Kathy determined/that the/program/hay 
not been in operation long enough to 
reach a conclusion on thiy question yet. 
She/ acquired/ test scores' and/ gradey 
from/the/ district and/ widtrach scores' 
annually over the/ next 5 yeary. 


I mp roved/ dropout: 
and/ attendance/ 
rates' (W id the/ new 
technology and/ 
traininghelp 
lower the/ dropout: 
rate/ and/ increase/ 
attendance/?) 


• Dropout rate/ 

• Attendance/ rate/ 


Again, Kathy felt that more/ time/ way 
needed/to answer thiy question She/ 
coUected/the/ information from/ district 
recor dy and/ w id continue/ to erarvvine/ 
these/ recordyonan annual hasty. 


Cost~effectivenesy 
(ly thiy program 
ay or more/ cost - 
effective/ than 
other programy 
that may show 
Similar resulty?) 


• program/budget 

• budget of similar 
program/ 

• program/ 
evaluations' 


Kathy gotbudgetyofS programy she/ 
thought were/ similar in ter my of 
erpected resulty. She/ ranked/them/ ad 
by cost and/ expected/ resuXty. The/ cost of 
the/Rivery technology program/ ranked/ 
inthe/ middle/. It iy too- soon to draw 
conclusions' about result however. 


Effective/ 
professional 
development 
activities' (W id the/ 
professional 
development help 
teacher y integrate/ 
technology into 
their teachings?) 


• Review of teacher 
lesson plany 

• Observation of 
classrooms' 

• Teacher Survey 


Kathy both reviewed/ teacher y lesson 
plany and/ observed classes'. Kathy 
determined that teacher y were/ 
incorporating^ technology into- their 
teaching' more/ often after they had 
participated in professional 
development : She/ also surveyed 
teachery and deter mined that teacher y 
believedtechnology way improving 
their teaching, particularly in social 
Studies : 


Increased/ 
computer literacy 
(Will ytudenty 
and/teachery 
become/ more/ 
computer literate/ 
ay a/ result of the/ 
program/?) 


• Assessment of 
teacher computer 
literacy 

• Assessment of 
Student computer 
literacy 


Onboth teacher and student survey y, 
there/way a self- assessment of computer 
literacy. However, just OS' with test scores' 
and attendance/ rates', andbecause/ 
baseline/ data are/ only now being 
collected, Kathy feely more/ time/ Oy 
needed to reach a conclusion OS' to 
whether there/ ly change/ over time/. 
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: "What Are My Conclusions? " Worksheet 



Try to fill out the same form that Kathy did: 



Goal 


Measure(s) 


Conclusion Process 
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How Do I Communicate My Results? 



Requirements 



First, figure out what you may be required to do. Many grants come with 
formal evaluation requirements that specify the types of reports that must be 
submitted, as well as when they are required. For example, one of the 
requirements of the Technology Literacy Challenge Fund is that districts 
receiving funding need to submit a report to their state annually. Take 
advantage of these requirements. Treat them as your chance to tell those in 
charge about your program and either why it is a success or what you would 
need to make it a success (e.g., more time, more resources). 



Next, figure out who your audiences are. You know you have to write a 
AUDIENCE couple of reports - or at least a couple of memos. Your next step is to figure 

out who will be reading them. Make a list of (1) all those you are required to 
communicate your results to, and (2) all others whom you want to know about 
your evaluation. 



Then, for each audience, ask yourself these questions: 



• What background do they have in the subject or regarding the program? 

• What will they want to know? 

• How much time and/or interest will they have? 



This will help you tailor your communication. In general, if you are given 
guidelines, try to stick to them as closely as you can. If you aren't given 
guidelines, then make your best guess. These are some tips to keep in mind: 

• If the audience already has background information on the program, try to 
stick as much as you can to providing only specific findings from your 
evaluation. If not, then you may have to try to “bring them up to speed." 

• Figure out which of your goals addressed what they wanted to know and 
why they wanted the evaluation. Address these goals. 

• If the audience wants something immediately, write a short memo 
summarizing major findings first, and then follow it up with a longer report. 

• Don't be afraid of including recommendations or possible areas of change. 
If you want to make changes, you're going to have to talk about it sooner 
or later, and having it in a report is a way to start the conversation. 



There Are 
Many Ways to 
Communicate 



Long reports are not the only way to communicate results. Reports are 
only one way to communicate results. Other options (some may be required, 
some not) include: 



9 A memo or letter 
® A phone call 

• A presentation before a board or 
committee 



A publication in a journal, 
newspaper, or magazine 
A workshop for teachers 
A webpage 
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How Did Rivers CoMatuiNicATE Results? 






8f€i 



Earlier, Kathy had listed the major stakeholders in the program. They included: Mr. Jones, the 
high school principal; Dr. Walker, the superintendent; Ms. Phillips, the professional development 
coordinator; teachers; and parents. In addition to this list, Kathy added the school board and Ms. 
Johnson, head of the Information Management Systems Department. She felt that the school 
board would be responsible for providing more funding for technology, so they should know the 
results, and Ms. Johnson requested to see any results as they would affect future equipment 
funding, and thus, information systems resources. 

Next, Kathy brainstormed about how to "spread the word" to all of these people. Mr. Jones was 
mostly interested in test scores and attendance and dropout rates. Dr. Walker would be interested 
in the same thing, but also would be interested in her cost comparison of Rivers’ program with 
others. In addition, both Mr. Johnson and Dr. Walker might be interested in any anecdotal 
evidence of teacher or student use. Ms. Phillips and teachers would probably want information 
from the surveys about the professional development. Parents would probably be interested in the 
self-assessments of computer literacy and test scores. They also might be interested in any 
scenarios that Kathy can provide about how their children are using computers or how teachers 
are using computers in the classrooms. Oh, and last but not least, the school board probably 
would be interested in something similar to what the superintendent was getting, and Ms. Johnson 
would probably be interested in any anecdotes or information about equipment. From this, Kathy 
listed what types of communication she thought she could produce (e.g., reports, presentations), 
and to whom they would go. 



“Report, Pretentetftorii eta. ".■■■■ y,' : l .1 


To- whom/ it would/ ao- 


Short report on iaUifoctCon 


— <2 

Joney, Walker, School Board/ 


ktemo- on reading' te^tyorey and/ 
attendance/ dropout conclu&iony 


Joney, Walker 


Report on impact of professional/ 
development 


PhULipy 


Short report on Stflf-aweiymenl of 

computer literacy 


Joney, Walker, School 'Board/ 


(Dvervcew atPTA meeting/ (computer 
Uteraoy, yati^factLon, anecdotey) 


Everyone/ 


cvt Scbxyxyi/'Box^rd/ Fneettn^ 


SchoolBoard 


Mennoto- My. Johnson re/: equipment 
anecdotey (also offer her copiey of reporty) 


Johnson 



It looked like a lot of work, but Kathy kept reminding herself that "short" report meant just that - 
short. She realized that, first of all, she didn't have time to write anything extensive, and, second, 
the people she was giving it to (e.g., the principal and superintendent) didn't have time to read 
anything extensive. However, it was still going to take some time to write and, since the school 
had spent quite a lot of money, the principal and superintendent wanted some feedback ASAP. 
So, before she got to work, Kathy decided to send them both an e-mail message giving a few 
highlights of what she found (e.g., a few descriptions of lessons she received from the survey), 
and telling them what kinds of reports, etc., she was planning. They were both satisfied with this 
and it gave them something to tell the school board. 
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How Do I Communicate My Results?" Worksheet 






What requirements do you have? 



To whom do you need to communicate findings? 

Name Type of Information 



What reports or memos could you write and to whom would they go? 

Report or Memo 



Recipient 



What presentations could you do? 

Presentation 



Recipient 



Is there anything you need to do in the meantime before the reports or presentations (e.g., e- 
mails)? 



AZ 
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Where Do I Go From Here? 



What You and 
Others Want 



Fostering 

Change 



Some 

Changes Take 
Time 



ij Follow-up 
* Research 



You've completed a lot of work. People have reports sitting on their desks. 
You've given presentations. What’s the next step? How do you help make 
sure that the reports don't just sit there collecting dust or that people don't 
forget about the presentation five minutes out the door? How do you help 
make sure that your information is used? 

First, think about what changes you and others would like to see. 

Before you can attempt to persuade others to use your information, you need 
to figure out what you would like to happen. What changes would you like to 
see or what decisions do you think need to be made as a result of your 
information? Then, think about what changes others might want and/or how 
they might want to use the information. This will provide two benefits. First, if 
people have the same objective in mind (e.g., "we need to get a full-time 
technical support person at the school"), they can work together. Second, 
knowing how others would like the information used gives you more 
awareness of where they are coming from and, thus, more information on 
how they would best be motivated. 

Next - take action! Have you ever heard the expression, "the squeaky 
wheel gets oiled"? This is your chance to be a squeaky wheel. You have 
evidence from your evaluation, you’ve given reports to people, you know what 
you want done. Ask for it! Find out who is in charge of making the changes 
you want. Find out if he/she has had a chance to read your report. If you 
have made recommendations in the report, you can bring this up (e.g., “I was 
curious to know your thoughts on ....” or “What did you think about ...?'') 
Otherwise, just bite the bullet and bring it up. Tell the person in charge that 
you have a suggestion and want to know what he or she thinks about it. 
However, be cautious. You have already had a chance to process the 
information that you've collected, but others have not. If they need a little 
breathing space to think about it and process it, by all means, give it to them. 

Know that some changes take time. It's tempting to want to see results 
right away, but, in most cases, it won't happen. Change, especially large- 
scale change, takes time. People need to process the information you've 
collected, figure out for themselves what they want to do about it, get 
motivated to do it, and then go through the appropriate process (e.g., get 
approvals, order equipment through procurement) before you will see any 
results from your evaluation work. Be patient. If you keep at it, change will 
happen, but it doesn't happen overnight. 

Finally, remember to follow up. Chances are that there are some questions 
that you were not able to answer at this point in time. You needed more 
information, more time, or both. Remember to come back to these items and 
"finish the job.” 
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How Did Rivers Use the 



Information? 




Kathy looked at the reports she had written to review the suggestions she had made. She also 
reviewed the notes she had taken when she surveyed a group of teachers about the professional 
development activities and the narrative comments that the teachers provided on the survey. The 
three areas where she saw potential for using the information for immediate change were as 
follows: 



• Although/Kathy wctyunMillOng/to-draw conclutiony from the/ daXzv on/ ytudent test 
scorey, she/ did/ note that studerdfr irv two- (dxvsse* whose readlr^perform/vnce/had/ 
been/parttcularly low theprevicnwy year seemed/ to- improve slightly mares thaw; 
Studenty whoseperformance had/ been/ more/ average'. 

• Tetudvery indicated/ that they could/benefit even/ more irom/theprofesstonal/ 
development they had/poAtictpated/ in/ if there were a/person/ on-site who- coidd/ 
cusslstthem/ with/ software use and/ integrating technology into- their lessony. 

• On/ self-assessment, both studenty OAxd/teabheryseerned/ comfortable using- some 
softwareprogramy but not othery. for example/, people / expressed/ comfort with 
searchengine&fortheWeb, but didrit seem/to-know about the/ CD -ROM resource* 

in/ the/ library. 



From these three findings, Kathy had three suggestions she could make to improve the program. 
First, she talked to Mr. Jones and Mr. Walker about the possibility of providing one or two 
additional computers in classes that enrolled large numbers of at-risk students. She also 
suggested offering teachers in these classes additional professional development. Since these 
students had shown some improvement and since the school board was particularly interested in 
trying to improve the performance of students performing at the bottom quartile on state tests, Mr. 
Jones and Mr. Walker agreed to include these expenses in the budget. When Kathy relayed 
teachers’ desire for technology assistance on-site, Ms. Phillips informed her of several funding 
opportunities that could support this type of staff. 

While Kathy was talking to Ms. Phillips, she showed her the results of the self-assessment and 
asked if there was a possibility of adding a training session on the CD-ROMs or replacing a 
session on use of the Internet with one on the CD-ROM software. Ms. Phillips was very glad to 
see the self-assessment information, agreed with Kathy's interpretation of a need for the class, 
and found a few more areas where training might be useful. She didn't think there would be any 
problem getting the school librarian to lead a session or two on the CD-ROM software and said 
that, depending on the topic, it might be easy to add other sessions. In any case, the self- 
assessment information would come in handy for the planning of next year’s professional 
development offerings. 

Finally, Kathy put notes in her files of where she still needed to collect information. Specifically, 
she needed to continue to monitor test scores and to collect further information on attendance and 
dropout rates. She had told her principal and superintendent that she would track these things 
and they would probably remind her, but putting it on a list of periodic data to collect would keep 
her from being caught by surprise when asked for th^ information. 
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1::" j" “Where Do I Go From Here?” Worksheet. 









What changes do you want? 







What changes do others want? 






Whom can or did you talk to about these suggestions? 






What did you find out could be done? 








What information do you need to follow-up on? 
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Additional Sources 






Bodily, S., & Mitchell, K. J. (1997). Evaluating challenge grants for technology in education: A 
sourcebook. Santa Monica, CA: RAND. 

Fink, A. (1995). A survey handbook. Newbury Park, CA: Sage. 

Frechtling, J. (ed.) (1996). User-friendly handbook for project evaluation: Science, mathematics, 
engineering, and technology education. Arlington, VA: National Science Foundation. 

Herman, J. L. (ed.) (1987). Program Evaluation Kit. Newbury Park, C A: Sage. 

International Society for T echnology in Education. (1 998). International Society for Technology in 
Education: Teachers helping teachers use technology in the classroom. 
http://www.iste.org. 

Morgan, D. L. (1997). The focus group guidebook. Newbury Park, CA: Sage. 

Rossi, P. H., & Freeman, H. E. (1993). Evaluation: A systemic approach. Newbury Park, CA: 
Sage. 

Soriano, F. I . (1995). Conducting needs assessments. Newbury Park, CA: Sage. 

U.S. Department of Education. (1998). U.S. Department of Education: Office of Educational 
Technology. http://www.ed.g 0 v/T echnology. 
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Appendix B - 
Worksheets 



For your convenience, we are including copies of all of the worksheets found throughout the 
Handbook. 
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"Why Am I Evaluating" Worksheet 

To answer the larger question of "Why am I evaluating?" you will need to consider the different 
stakeholders in your program and what they want out of the evaluation. 

Stakeholders What they want from an evaluation 



What do you want from an evaluation? 



Looking at what both you and other stakeholders want from an evaluation, what are two or three 
major reasons or purposes for doing your evaluation? 



How might these reasons affect the design of your evaluation? 



rr 
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j "Where Do S Start? " Worksheet 1 

What documents are available (e.g., state and/or local technology plans)? 




Ask questions about the plans. What major stakeholders were involved in creating the 
documents that were collected? Why were the plans written? Are the state and local plans similar 
(e.g., have similar timeframes)? Are the plans being implemented? How have plans changed over 
the years? What are some major goals in the plans (e.g., "use technology to improve student 
writing skills")? 








How is technology integrated into the curriculum and/or standards? 




Whom could you talk to about the evaluation? 




Who else may be collecting data you can use? 


Are there any other resources that you learned about by talking to people? 




What does the Internet have to offer? 
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"What Questjons Should I Ask?" Worksheet 

What do you want to know? (You may want to look back to the worksheet on page 5 for ideas on 
why you are evaluating and what you want from an evaluation.) 



What will others want to know (e.g., principal, superintendent, board members, funding agency, 
teachers, parents, students)? 



What other requirements are there? 



After this exercise, what are your “top 5” or "top 1 0" goals and questions? 

Goals Research Questions to Address Goals 
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"What Information Do II Meed to Collect?" Worksheet * 

wnr n k 9 thrlnh a th Tj. ' r6Search questions you came up with in the last section as a starting point, 
result ^ ^ he ° ° Wing 9nd - YoU may have t0 revise Some of your goals and questions as a 



Goals 


Indicators 


| Benchmarks 


J Measures 
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"What Are My Conclusions? " Worksheet 

Try to fill out the same form that Kathy did: 



Goal 


Measure(s) 


Conclusion Process | 






































< 
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How Do l Communicate My Results?" Worksheet 



What requirements do you have? 



To whom do you need to communicate findings? 

Name Type of Information 



What reports or memos could you write and to whom would they go? 

Report or Memo 



Recipient 



What presentations could you do? 

Presentation 



Recipient 



Is there anything you need to do in the meantime before the reports or presentations (e.g., e-mails)? 
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“Where Do I Go From Here?” Worksheet 

What changes do you want? 



What changes do others want? 



Whom can or did you talk to about these suggestions? 



What did you find out could be done? 



What information do you need to follow-up on? 
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Appendix C - 
Examples of Technology Surveys 



The U.S. Department of Education, through the American Institutes for Research, has obtained 
permission from copyright holders and other producers to reproduce the questionnaires in this guide. 
Permission has also been granted for the reader to use the questionnaires, or parts of the 
questionnaires, for research purposes. Permission to publish surveys, however, only extends to this 
Handbook and not to any other publications. 



Surveys included represent examples of instruments addressing different issues and populations. Surveys 
selected include only a small portion of those available for use. 
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Robert E. Bartman 

Commissioner of Education 




Missouri Department of Elementary and Secondary Education 
P.O. Box 480 

Jefferson City, Missouri 65102-0480 



Department of Elementary and Secondary Education 
District Computing Survey 

To more clearly understand the status of educational computer technology in Missouri’s public 
schools, the Department of Elementary and Secondary Education (DESE) is sponsoring a Census 
of Technology in the Public Schools. The Office of Social and Economic Data Analysis 
(OSEDA) from University of Missouri Outreach/Extension and MOREnet are cooperating to 
conduct this census. 

Please complete the census form contained in this document to reflect district status as of 
February 1, 1997. Return the census form in the enclosed preaddressed, prepaid envelope by 
February 14, 1997. When the statewide census is completed, summary data will be available on 
the DESE Web site (http://services.dese.state.mo.us). 

Thank you for your cooperation. 



First, please complete the census-taker information box below. 
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2 . 



3. 



4. 



Status as of February 1, 1997 

Including part-time staff, how many staff are employed by your district specifically to support 
instructional telecommunications and computing? (FTE means Full Time Equivalent - for 
example, two half-time staff equal one FTE.) 

Total FTE 1 I 



Does your district support a Wide Area Network? 



Yes 



How many Local Area Networks (LANs) are supported in your district? 

Number of LANs I 1 




Please check the type(s) of operating system(s) used with your LAN(s). Check all that apply. 



Operating System Number in Use and Version 



Novell 




LANtastic 




Windows NT 




AppleShare/ 

Macintosh 




Other 





5 . Has your district adopted a district technology plan? 

Yes L I No 1 1 

6. Has your district adopted a district Acceptable Use Policy? 

Yes f I No I I 

7. Does your district have a Web page? 

Yes [" I No 1 1 

8. If your district has a Web page, what is the URL? 

URL 
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9. Next, for each group below please record the number of accounts provided by your school district 
to different types of users of the specific services provided: 

* Dialup Internet Access - Access through a district-supported modem pool. This dialup access would be to 
a local phone number associated with your school district and not the DESE modem pool in St. Louis or 
Kansas City. 

* Individual E-Mail - These e-mail accounts are not part of the DESE Project. They will be in the form of 
userid@building.district.lc 1 2.mo.us 



Services Provided 



Type of User 


Number of 
Dialup 
Internet 
Access 
Accounts 


Number of 
Individual 
E-Mail 
Accounts 


Teachers 






Students 






Administrators 






Local Patrons 






Parents 






Others 







10. What are the top three instructional technology support services that are not adequately provided 
in your district? Typical instructional technology support services might include maintaining a 
LAN, setting up a desktop computer, providing training on using the computer, installing software 
on a file server, or providing “help desk” question-answer services for teachers and students. 
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Robert E. Bartman 

Commissioner of Education 




Missouri Department of Elementary and Secondary Education 
P.O. Box 480 

Jefferson City, Missouri 65102-0480 



Department of Elementary and Secondary Education 
Building Computing Survey 



To more clearly understand the status of educational computer technology in Missouri’s public 
schools, the Department of Elementary and Secondary Education (DESE) is sponsoring a Census 
of Technology in the Public Schools. The Office of Social and Economic Data Analysis 
(OSEDA) from University of Missouri Outreach/Extension and MOREnet are cooperating to 
conduct this census. 

Please complete the census form contained in this document to reflect building status as of 
February 1, 1997. Return the census form in the enclosed preaddressed, prepaid envelope by 
February 14, 1997. When the statewide census is completed, summary data will be available on 
the DESE Web site (http://services.dese.state.mo.us). 

Thank you for your cooperation. 



First, please complete the census-taker information box below. 



Building.Computing Contact: 



Name 



E-mail Address 



Telephone Number 



Fax Number 



Building 



District 



Building Address 
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Status as of February 1, 1997 



Please determine how many desktop computers there are in the following settings or 
locations in your building, and of these computers, how many are Internet capable. 



In Co l umn A please record the total number of desktop computers in your building by type of 
setting or location. Where a setting has multiple uses, choose one use to avoid double counting. 

In Column B please record the number of those desktop computers that are Internet capable . 

Internet capable means either 

* PC’s with at least a 386 processor with at least 8 meg of memory running windows, or 

* Mac's with at least a 68030 processor with at least 8 meg of memory. 



Table 1 



Column A Column B 



Settings/Locations 


Total 

Computers 


Internet Capable 
Computers 


Admin/Business Office 






Computer Labs 






Classrooms 






Library/Media Ctrs 






Voc/Tech Area 






Other Settings or 
Locations in the Biding 







We now would like you to report the number of Internet capable desktop computers 
according to network Status. For each setting below, please record the number of these 
desktop computers that are on a Local Area Network (LAN). (A computer is networked 
if it is on a LAN with or without a file server.) 

Table 2 



Settings/Locations 


Number of 
Networked Internet 
Capable Computers 


Admin/Business Office 




Computer Labs 




Classrooms 




Library/Media Ctrs 




Voc/Tech Area 




Other Settings or 
Locations in the Biding 
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3. Next, for each setting below please record the number of Internet capable desktop 

computers by type of Internet access . In terms of Internet access, please classify your 
Internet capable desktop computers into the following three mutually exclusive groups: 

* No Internet Access - The computer is unable to access the Internet because it lacks necessary hardware, 
software, line connection, or Internet account 

* Dial-up Only - The computer is able to access the Internet because it has the necessary hardware, 
software, line connection, and Internet dial-up account 

* Dedicated Line - The computer is able to access the Internet because it has the necessary hardware, 
software, and access to a dedicated line connection. (If a computer has both dial-up capability and access to 
a dedicated line connection, count the computer as a dedicated unit only.) 



Table 3 



Settings / 
Locations 


Number of 
Internet Capable 
Computers (Table 
I, Column B) 


Number with No 
Internet Access 


Number with 
Dial-up Only 


Number with 
Dedicated Line 


Admin/Business Office 










Computer Labs 










Classrooms 










Library/Media Centers 










Voc/Tech Area 










Other Settings or 
Locations in the Biding 




' 







4. For your Internet capable desktop computers only, please record the number of desktop 
computers in your building by type of operating system . 

Table 4 



Operating System 


Number of 
Internet Capable 
Computers 


Windows 3.1 




Windows 95 




Windows for Workgroups 




Windows NT Workstations 




Macintosh 




Other 
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5. If your building has a dedicated line connection, what speed is the connection? 

Speed of the dedicated connection I I 

6. Who is your Internet service provider? 

MQREnet/DESE Technology Network Project I I 
Other (please list) . 

7. Is this building on a Wide Area Network? 

Yes I I No 1 1 

8. Does your building have a Web page? 

Yes [ 1 No 1 1 

9. If your building has a Web page, what is the URL? 

URL 

10. How many classrooms remain to be wired for the entire building to have network access 
to the Internet? (For example, 10 of 20 classrooms.) 

Number of classrooms remaining to be wired 1 1 

of 1 I total classrooms in the buil ding 

1 1 . How many additional computers would be required to provide at least one Internet 
accessible desktop computer per classroom? 

Number of additional computers required 1 I 
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School Hardware Resource Survey 



School: 

District: 



Type: High Mid Elem 



SCHOOL WIDE INFORMATION 



Number of Buildings 




Ethernet Hubs 




• with Acceptable Electrical Power 




Routers 




• on Network 




Multiplexers (Newbridge) 




Rooms Served by Network 




PBX/Key Systems 




1 Communications Closets (IDF/MDF) 




Main Data Lines to Site 





EQUIPMENT AND DEVICES IN ROOMS 
Total Number of Areas and Rooms 



E Dmin 

□PEN SYSTEMS 



www.edmin.com 



(0 


<0 


E 


05 

Q> 
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o 
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k. 

C0 


<D 


CO 


O 


CO 


if* 


o 


O 



(0 

a> 

u 

as 

A 

□ 



a> 
o w 

3 E 

g g 

4 “ 



CO 

a> S 

o< 
as ^ 

0) j- 

o 



Computers 

Mainframe Terminals 



Description 



Vintage I: Other 
Vintage I: Apple 



Tandy, Commodore 64, etc. 



Apple lie, Apple ligs, Apple 114- 



Vintage II: Apple (68000/020/030) 
Vintage III: Apple (68040) 



Mac LC, LCII, LCIII, lici. Plus, SE, etc. 



Quadra 605, Centris 610, LC 475, etc. 



Vintage IV: Apple (Power Mac) 
Vintage I: PC (8086/8088/286) 



Performa 6400, 6100, 7200, 8500, 4400, etc. 



286 



Vintage II: PC (386) 
Vintage III: PC (486) 



386dx, 386sx 



486dx, 486SX 



Vintage IV: PC (Pentium) 
CD Capable Computers 



Pentium, Pentium Pro, Pentium II 



Equipped with internal/external CD-ROM drive 



AV Capable Computer 
Portable Computer 



Equipped with audio/video input/output 



Other Computer Devices 



Laptop, notebook, Powerbook, etc. 



LCD Panel, LCD Projector 
Laser Printer 



Description 

Attaches to a computer to view screen 



Dot Matrix Printer 
Inkjet Printer 



CD-ROM Tower 
Scanner 



Alpha Smarts/Word Processor 
1 Digital Camera 



Video Editing System 
Presenter 



LTV, etc. 



N etworking Equipment 

Vintage I: Network Wiring 



Local i alk/Baseband 



Areas Served 



Vintage II: Network Wiring 



Vintage III: Network Wiring 



Ethemet/Token Ring 



Fast Ethemet/FDDI 



Vintage I: File Sharing 



Networking — Peer to Peer 



Vintage II: File Sharing 



Vintage III: File Sharing 



Networking — Dedicated File Server 



Networking — NOS Server (Novell, Win NT) 



Modems 



Dedicated Modem Lines 



Speeds: 



Dedicated Internet Access 



WAN Access 

Form: HWRS ver2.2mm 2/3/98 



Hi 
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School: Type: High Mid Elem 

District: Date of Survey: 


<0 

E 

o 

o 

<0 

<0 

05 

o 


(0 

CO 

0> 

w> 

< 

o 

£ 

o 


<0 

.2 

z 

C9 

£ 


0) 

o m 
3 E 

o O’ 

ui o 

® DC 
EE 


Teacher Work 
Areas 




Broadcasting Equipment 


Description 




■ 








TV Broadcasting System 














Voice Broadcasting System 














: Audio/Visual Equipment 


Description 












Phones 














Dedicated FAX lines 














Stereo System 


Component, Boom Box, etc. 












Cassette Deck 














TV’s 














VCR’s 














Laserdisc Player/DVD Player 














Projectors 














Video Conferencing Links 














Video Retrieval Services 














Cable Access 














Satellite Access 














Video Camcorder 














Other Equipment (write in below) 
























































Future Plans 


-L Notes 


(estimate numbers) 


Computers 














Network Connected Computers 














Phones 














Dedicated FAX Lines 














Dedicated Modem Lines 














Video Conferencing Links 














Video Retrieval Services 














Dial-up Internet Connections 














Direct Internet Connections 














: Other Equipment (write in below) 






• ! ’;h ..•)/ 




















































































































































Does your site have acceptable electrical power for your planned computers? Yes No — 
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Room Hardware Resource Survey 



School: 

District: 



Room Number: . 

Date of Survey: 



E Dm in 

□PEN SYSTEM S 



www.edmin.com 





TYPE OF ROOM 


~ 




Classroom Office Area 


Library Resource Room/Lab Teacher Work Area 




EQUIPMENT AND DEVICES IN ROOMS 


Computers 


Description 


Number 



[^aMrame Terminals 






Vintage 1: Other 


Tandy, Commodore 64, etc. 




Vintage 1: Apple 


Apple lie, Apple ligs, Apple 11+ 




Vintage II: Apple (68000/020/030) 


Mac LC, LCII, LCIII, lici, Plus, SE, etc. 




Vintage III: Apple (68040) 


Quadra 605, Centris 610, LC 475, etc. 




Vintage IV: Apple (Power Mac) 


Performa 6400, 6100, 7200, 8500, 4400, etc. 




Vintage 1: PC (8086/8088/286) 


286 




Vintage II: PC (386) 


386dx, 386SX 




Vintage III: PC (486) 


486dx, 486SX 




Vintage IV: PC (Pentium) 


Pentium, Pentium Pro, Pentium II 




CD Capable Computers 


Equipped with internal/external CD-ROM drive 




A V Capable Computer 


Equipped with audio/video input/output 




Portable Computer 


Laptop, notebook, Powerbook, etc. 





Other Computer Devices 


Number 




1 Broadcasting Equipment 


Number 


LCD Panel, LCD Projector 




TV Broadcasting System 




Laser Printer 




Voice Broadcasting System 




Dot Matrix Printer 








Inkjet Printer 




Audio/Visual Equipment 


Number 


CD-ROM Tower 




Phones 




Scanner 




Dedicated FAX lines 




Alpha Smarts/Word Processor 




Stereo System 




Digital Camera 




Cassette Deck 




Video Editing System 




TV’s 




Presenter 




VCR’s 








Laserdisc Player/DVD Player 




Networking Equipment 


1 Number 


Projectors 




Video Conferencing Links 




Vintage 1: Network Wiring 




Vintage II: Network Wiring 




Video Retrieval Services 




Vintage III: Network Wiring 




Cable Access 




Vintage 1: File Sharing 




Satellite Access 




Vintage II: File Sharing 




Video Camcorder 




Vintage III: File Sharing 








Modems 




Other Equipment (write in below) 


Dedicated Modem Lines 








Dedicated Internet Access 








WAN Access 
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Developed as part of: 



The Evaluation of the 
Technology Literacy Challenge Fund 



Developed by: 

American Institutes for Research 
1000 Thomas Jefferson Street, N.W. 
Suite 400 

Washington, D.C. 20007 
(202) 944-5300 

Rita Kirshstein, Project Director 



1998 
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Please provide the following information: 



Name of the school 

School address 

School phone number 



1 . What level is your school? Please check one. 

Elementary (includes at least grades 1-4) □ 

Middle/Intermediate/Junior (includes no grade below 4 or 

or above 10, but includes at least grade 7-8) □ 

High/includes at least grade 1 1 and 12) □ 

Combined (combination of one or more of the above categories) □ 



2. What was the total enrollment in your school on October 1 , 1998? 



3. How many classroom teachers were employed full-time at your school on October 1 , 1998? 



4. How many classrooms/instructional rooms does your school have? 



5. How many computers does your school own (include portable computers and laptops)? 

a. For staff/administrative use only (no student access) 

b. For student use (include any used by students, 

even if staff members also use them) 

c. Other ( please specify ) 

d. Total ( should equal the sum of a, b, and c) 

6, How many computers are located in each of the following areas? 

74 

a. Classrooms/Instructional rooms 
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b. Libraries/Media Centers 

c. Computer labs 

d. Staff/administrative offices 

e. Other {please specify ) 

f . Total ( should equal the sum of a through e) 



7. Does your school have a local area network? ( please check one) Yes □ No □ 



8. How many of your computers are connected to this network? 



9. How many classrooms/instructional rooms have at least one computer permanently located in 
them? 



10. How many classrooms/instructional rooms have at least one computer permanently located in 
them that is connected to the Internet? 



1 1 . Of those computers that are for student use, please indicate the number you have of each brand 
and capacity: 

a. IBM-compatible 286 

b. IBM-compatible 386 

c. IBM-compatible 486 

d. IBM-compatible Pentium 

e. Other IBM-compatible 

f. Apple II 

g. Macintosh 

h. Power Mac 

i. LC 475/LC 580 

j. Other Apple or Macintosh clone 

k. Total (should equal #5h above) 
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12. How many of these machines have each of the types of hardware specified available for student 
use? 

a. CD-ROM drive/DVD drive 

b. Microphone 

c. Speakers 

d. Interactive videodisc player 

e. Video camera 

f. Scanner 

g. Printer 

h. Other (please specify ) 

Other (please specify ) 

Other (please specify ) 

1 3. What types of software are available to students (please check those that are available) 



a. Word processing □ 

b. Spreadsheet □ 

c. Database □ 

d. Graphical □ 

e. Desktop publishing □ 

f. Presentations □ 

g. On-line services (e.g., America Online, 

CompuServe, KidsNet, or Prodigy) □ 

h. Hypermedia/Multimedia (e.g., HyperCard) □ 

i. Integrated Learning Systems (e.g., Jostens, CCC) □ 

j. Simulation Programs □ 

k. Drill/Practice Programs/Tutorials □ 

l. Other (please specify ) □ 



14. How many teachers received school or district-sponsored training in the use of computers during 
school year 1 997-98? 



1 5. Does your school have a technical support person available on staff to provide assistance to 
teachers in using the computers? ( please check one) 



Yes, full-time 


□ 


Yes, part-time 


□ 


No 


□ 



16. To how many professional journals on educational technology or that include articles on 
educational technology does your school subscribe? ( please check one) 

None □ 1 □ 2-3 □ 4 or more □ 
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17. Is there any reading material available in your school for teachers regarding technology or the 
integration of technology into the classroom? ( please check one) 

Yes □ No □ 



1 8. Does your school follow a written curriculum? ( please check one) 

Yes □ No □ (SKIP TO Q. 20) 

19. If yes, is technology specifically mentioned in the curriculum, either as a tool for learning or as a 
separate subject? (please check one) 



Yes, as a tool for learning 


□ 


Yes, as a separate subject 


□ 


Yes, both 


□ 


No 


□ 



20. Do you have minimum computer technology standards or requirements for teachers or students 
to meet? 



Yes, for teachers only □ 

Yes, for students only □ 

Yes, for both teachers and students □ 

No □ 
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Teacher Surveys 
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Technology Needs Assessment Survey 



Name: 



School: 



District: 



Type: certified classified 
Grades: 



Position: 



E Dmin 

□ PEN SYSTEMS 



www.edmin.com 



each item. 



YOUR TECHNOLOGY BACKGROUND 


Have vou used the following technologies? 


A = yes, frequently B = yes, rarely C = no D = don ’t know 


A 


B 




C 


1 ° 


E 


1. IBM-Compatible Computer 


Windows 3.11, Windows ’95, Windows NT 












2. Apple Macintosh Computer 


Performs, PowerMac, LC, Quadra, 












3. Other Computer 


DOS, Apple He, UNIX, etc. 










•I:-".:?: : ;i 


4. VCR 


Recorder, player 












5. Laserdisc Player or DVD Player 














6. Video Camcorder 












ill® 


7. Digital Camera 












iifell'fl 


8. LCD Panel or Computer Projector 












yjj# 


9. Internet Web Browser/Online Service 


Navigator, Internet Explorer, AOL, Prodigy, etc. 










Ipir: 


10. Fax Machine 














11. Modem 














12. CD-ROM 












■r'i'-V 


13. Scanner 














14. Networks (File/Print Sharing) 














STA 


FF CENTERED TECHNOLOGY 


Have you used computer technology for the following for your job? 


A = yes, frequently B = yes, rarely C = 


-no D = don’t know 




A 


B 


C 


JD 


E 


15. Student Management 


Grading/attendance/assessment programs 












16. Student Information 


Student records, discipline, health systems 












17. School Management 


Budget, personnel, scheduling/calendar, etc. 










i A' \ 


18. Word Processing 


Word, Word Perfect, ClarisWorks, Simpletext 












19. Spreadsheets or Databases 


Excel, Filemaker Pro 












20. Desktop Publishing 


Pagemaker, QuarkXPress, Print Shop Deluxe 










V”. 


21. Authoring or Multimedia 


HyperStudio, Director 










!■' •:® 5 


22. Instructional Demonstration/Tutorial 


PowerPoint, Persuasion 










l|i:i:| 


23. Information Retrieval 


Infotrac, SIRS, Library Circulation, etc. 










rV-i 


24. Audio/Video Capture or Digitizing 


Premiere, Videoshop 










,V- 


25. Art/Graphic Development 


Photoshop, Painter, Illustrator, Canvas 












26. Computerized Testing 


Microtest 










ilii 


27. Internet or Online Service Access 


Navigator, Internet Explorer, AOL, CompuServe 










’J: ... '! 


28. Web Page Development 


Frontpage, HomePage, Pagemill, BB Edit 












29. E-mail 


Eudora, Outlook, Exchange, Emailer 
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STUDENT-CENTERED TECHNOLOGY 


In your classes, do students use the following computer-aided instruction (CAD? 


A = yes, frequently B = yes, rarely C = no D = don’t know E = not applicable 


A 


B 


c 


D 


E 


30. CAI: drill and practice/tutorial 














31. CAI: simulation/educational games 


Sim City, Sim Life, Civilization 












32. Word Processing 


Word, Word Perfect, ClarisWorks, Simpletext 












33. Information Retrieval 


SIRS, Periodic Index, Encarta, Library Circulation 












34. Problem Solving 


Math Blaster, Oregon Trail 












35. Database/Spreadsheet use 


Excel, Filemaker Pro 












36. Internet Access 


Telecommunications, research, current events 












37. Student Portfolio Building 














38. Authoring/Multimedia Development 


HyperStudio, Director 












39. Desktop Publishing 


Yearbook, journalism, newspaper 












40. Electronic Presentations 


PowerPoint, Persuasion 












41 . Video Development 














42. Open Lab Access 














43. Web Page Development 














it tne following technologies were available, how would you rate the potential of each 
instruction or to support instruction? 


to in 


npac 




A = high B = somewhat high C = somewhat low D = low E = don’t know 


A 


B 


c 


D 


E 


44. individual Computer 














45. Network File Server 














46. Modem/Bulletin Board/Telecommunications 












— 


47. Teleconferencing/Interactive Video 














48. Educational TV-ITV/Videotaoes 














49. Internet Access 














50. Laserdisc/DVD Player 














51. Multimedia System/CD-ROM 














52. Electronic Library Access 














53. LCD Panel/Computer Projector 














54. Electronic Chalkboard 














55. Voice Activation/Touch Screen Input Systems 














56. Video Camcorder 














57. Expert Systems for Diagnostics 














58. School/District E-mail 














59. Fax Machines 














60. High Quality Printing (color/laser) 














61. Video Capture/Editinq Equipment 














62. Web Server/Home Page Development Software i 














63. CD-ROM Production 














64. Desktop Publishing Resources 














65. Scanners 














66. Digital Cameras 
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STAFF DEVELOPMENT ACTIVITIES 


Identify the quantity of trainina vou have received in each of the following areas. 


A = extensive B = lots C = some D = minimal E = none 


A 


B 


c 


•Or 




67. Advanced Input/Output Devices 


Scanner, digital camera, thermal printer 












68. Telecommunications 


Modem, e-mail, bulletin boards, internet access 












69. TV/Audio/Video 


Educational TV, Laserdisc, VCR, Camcorder, etc. 












70. Word Processing 














71 . Database/Spreadsheet 














72. Authoring/Multimedia Development 














73. Portfolio Development 














74. Networking/Network Management 














75. School Management 


Budgets, scheduling, personnel 












76. Student Management 


Grading, attendance, student records 












77. Desktop Publishing 














78. Interactive Video 














79. Curriculum Specific Applications 














80. Basic Operating System Techniques 


Mac OS, DOS, Windows 3.11, Win ’95, Win NT 












81. CD-ROM/Multimedia Applications 














82. Electronic Research 


Online, CD-ROM based, databases 












83. HTML/Web Page Development 














Identify the quantity of trainina you feel you need in each of the following areas. 






A = EXTENSIVE B =,LOTS C - SOME D = MINIMAL E = NONE 






.J/v, 


D 




84. Advanced Input/Output Devices 


Scanner, digital camera, thermal printer 












85. Telecommunications 


Modem, e-mail, bulletin boards, internet access 












86. TV/Audio/Video 


Educational TV, Laserdisc, VCR, Camcorder, etc. 












87. Word Processing 














88. Database/Spreadsheet 














89. Authoring/Hypermedia Development 














90. Portfolio Development 














91. Networking/Network Management 














92. School Management 


Budgets, scheduling, personnel 












93. Student Management 


Grading, attendance, student records 












94. Desktop Publishing 














95. Interactive Video 














96. Curriculum Specific Applications 














97. Basic Operating System Techniques 


Mac OS, DOS, Windows 3.11, Win ’95, Win NT 












98. CD-ROM/Multimedia Applications 














99. Electronic Research 


Online, CD-ROM based, databases 












100. HTML/Web Page Development 
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CODE 77 Self-Evaluation Rubrics for Basic Teacher Computer Use 



Please judge your level of achievement in each of the following competencies. Circle the number 
which best reflects your current level of skill attainment. (Be honest, but be kind.) At the end of the 
training program, you will complete the same set of rubrics that will reflect your level of skill 
attainment at that time. (Level 3 is considered mastery.) This tool is to help measure the 
effectiveness of our training program, and to help you do a self-analysis to determine the areas in 
which you should continue to learn and practice. Keep a copy of these rubrics to refer to during the 
training. 

I. Basic computer operation 

Level 1 I do not use a computer. 



Level 2 


I can use the computer to run a few specific, preloaded programs. It has little effect 
on either my work or home life. I am somewhat anxious I might damage the machine 


Level 3 


or its programs. 

I can set-up my computer and peripheral devices, load software, print, and use most 
of the operating system tools like the scrapbook, clock, note pad, find command, and 
trash can (recycling bin). I can format a data disk. 


Level 4 


I can run two programs simultaneously, and have several windows open at the same 
time. I can customize the look and sounds of my computer. I use techniques like 
shift-clicking to work with multiple files. I look for programs and techniques to 
maximize my operating system. I feel confident enough to teach others some basic 
operations. 



II. File management 

Level 1 I do not save any documents I create using the computer. 



Level 2 


I save documents I’ve created but I cannot chose where they are saved. I do not back- 


Level 3 


up my files. 

I have a filing system for organizing my files, and can locate files quickly and 
reliably. I back-up my files to floppy disk or other storage device on a regular basis. 


Level 4 


I regularly run a disk-optimizer on my hard drive, and use a back-up program to make 
copies of my files on a weekly basis. I have a system for archiving files which I do 
not need on a regular basis to conserve my computer’s hard drive space. 



TIT. Word processing 

Level 1 I do not use a word processor, nor can I identify any uses or features it might have 



Level 2 


which would benefit the way I work. 

I occasionally use the word processor for simple documents which I know I will 
modify and use again. I generally find it easier to hand write or type most written 
work I do. 


Level 3 


I use the word processor for nearly all my written professional work: memos, tests, 
worksheets, and home communication. I can edit, spell check, and change the format 
of a document. I can paginate, preview and print my work. I feel my work looks 


Level 4 


professional. 

I use the word processor not only for my work, but have used it with students to help 
them improve their own communication skills. 
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IV. Spreadsheet use 

Level 1 I do not use a spreadsheet, nor can I identify any uses or features it might have which 



Level 2 


Would benefit the way I work. 

I understand the use of a spreadsheet and can navigate within one. I can create a 
simple spreadsheet which adds a column of numbers. 


Level 3 


I use a spreadsheet for several applications. These spreadsheets use labels, formulas 
and cell references. I can change the format of the spreadsheets by changing column 
widths and text style. I can use the spreadsheet to make a simple graph or chart. 


Level 4 


I use the spreadsheet not only for my work, but have used it with students to help 
them improve their own data keeping and analysis skills. 



V. Database use 

Level 1 Ido not use a database, nor can I identify any uses or features it might have which 



Level 2 


would benefit the way I work. 

I understand the use of a database and can locate information within one which has 


Level 3 


been pre-made. I can add or delete data in a database. 

I use databases for a personal applications. I can create an original database - defining 
fields and creating layouts. I can find, sort and print information in layouts which are 
clear and useful to me. 


Level 4 


I can use formulas with my database to create summaries of numerical data. I can use 
database information to mail merge in a word processing document. I use the 
database not only for my work, but have used it with students to help them improve 
their own data keeping and analysis skills. 



VI. Graphics use - 

Level 1 I do not use graphics in my word processing or presentations, nor can I identify any 



Level 2 


uses or features they might have which would benefit the way I work. 

I can open and create simple pictures with the painting and drawing programs. I can 
use programs like PrintShop or SuperPrint. 


Level 3 


I use both pre-made clip art and simple original graphics in my word processed 
documents and presentation. I can edit clip art, change its size, and place it on a page. 
I can purposefully use most of the drawing tools, and can group and un-group objects. 
I can use the clipboard to take graphics from one application for use in another. The 
use of graphics in my work helps clarify or amplify my message. 


Level 4 


I use graphics not only for my work, but have used it. with students to help them 
improve their own communications. I can use graphics and the word processor to 
create a professional looking newsletter. 



VII. Hypermedia use 

Level 1 I do not use hypermedia (HyperStudio), nor can I identify any uses or features it 



Level 2 
Level 3 


might have which would benefit the way I work. 

I can navigate through a pre-made hypermedia program. 

I can create my own hypermedia stacks for information presentation. These stacks 
use navigation buttons, sounds, dissolves, graphics, and text fields. I can use an LCD 
projection device to display the presentation to a class. 


Level 4 


I use hypermedia with students who are making their own stacks for information 
keeping and presentation. 
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VIII. Network use 

Level 1 I do not use the on-line resources available in my building, nor can I identify any uses 



Level 2 


or features they might have which would benefit the way I work. 

I understand that there is a large amount of information available to me as a teacher 
which can be accessed through networks, including the Internet. With the help of the 
media specialist, I can use the resources on the network in our building. 


Level 3 


I use the networks to access professional and personal information from a variety of 
sources including networked CD-ROM reference materials, on-line library catalogs, 
the ERIC database, and the World Wide Web. I have an e-mail account that I use on a 


Level 4 


regular basis. 

Using telecommunications, I am an active participant in on-line discussions, can 
download files and programs from remote computers. I use telecommunications with 
my students. 



IX. Student Assessment 

Level 1 I do not use the computer for student assessment. 



Level 2 


I understand that there are ways I can keep track of student progress using the 
computer. I keep some student produced materials on the computer, and write 
evaluations of student work and notes to parents with the word processor. 


Level 3 


I effectively use an electronic grade book to keep track of student data and/or I keep 
portfolios of student produced materials on the computer. I use the electronic data 
during parent/teacher conferences. 


Level 4 


I rely on the computer to keep track of outcomes and objectives individual students 
have mastered. I use that information in determining assignments, teaching strategies, 
and groupings. 



X. Ethical use understanding 



Level 1 
Level 2 
Level 3 


I am not aware of any ethical issues surrounding computer use. 

I know that some copyright restrictions apply to computer software. 

I clearly understand the difference between freeware, shareware, and commercial 
software and the fees involved in the use of each. I know the programs for which the 
district or my building holds a site license. I understand the school board policy on 
the use of copyrighted materials. I demonstrate ethical usage of all software and let 
my students know my personal stand on legal and moral issues involving technology. 
I know and enforce the school’s technology policies and guidelines, including its 
Internet Acceptable Use Policy. I have a personal philosophy I can articulate 
regarding the use of technology in education. 


Level 4 


I am aware of other controversial aspects of technology use including data privacy, 
equitable access, and free speech issues. I can speak to a variety of technology issues 
at my professional association meetings, to parent groups, and to the general 
community. 
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CODE 77 Self-Evaluation Rubrics for Advanced Teacher Computer Use 
I. Instructional software use 

Level 1 I do not use instructional software as a part of my instructional program, nor am I 



Level 2 


aware of any titles which might help my students meet their learning goals. 

I use a few computer programs as an instructional supplement, as a reward, or with 


Level 3 


special needs children. 

I use several programs (drill and practice, simulations, tutorials, etc.) chosen by my 
department or grade level to help all my students meet specific learning objectives. 
The software allows me teach and/or reinforce concepts more effectively than 
traditional methods. When it is available, I use the software’s management system to 


Level 4 


help assess individual student performance. I use technological resources to meet the 
needs of students who do not respond to traditional methods of instruction. 

I seek out new programs for evaluation and adoption. I know sources of software 
reviews and keep current on new developments in computer technologies through 
professional reading and conference attendance. I share my findings with other 
professionals. 



II. Information literacy skills 

Level 1 I am not familiar with the term information literacy, nor do I know why such skills 



Level 2 


are important. 

As a part of my curriculum, I have library research projects and I support the library 
skills taught by the media specialist. I am aware that there are electronic resources 


Level 3 


available to my students. 

My curriculum includes multiple projects that have an information literacy 
component. These are team taught with the media specialist. I understand the Big Six 
or a similar information literacy process and design student projects so that they 
require higher level thinking skills, use electronic information sources, require the 
use of computer productivity software, and are authentically assessed. I guide my 
students in accessing, evaluating and using information and experts from world-wide 
sources through the Internet and video conferencing. 


Level 4 


I am actively involved in curriculum planning teams and advocate for 
multidisciplinary units and activities which require information literacy skills. I share 
successful units with others through print and electronic publishing and through 
conference presentations and workshops. 



m. Modification of instructional delivery 

Level 1 I have one or two effective methods of delivering content or teaching skills to my 



Level 2 


students. I do not use technology which requires that I change my instructional 
methodology. 

I have tried units or projects which are student-directed, use small groups, or are 


Level 3 


highly individualized, but I primarily use teacher-directed, whole group instruction. 

I use a variety of instructional delivery methods and student grouping strategies 
routinely throughout the year. I can design activities and approaches which both best 
fit the learning objectives and the availability of the technology available to me. I can 
use small groups working cooperatively or in rotation to take advantage of student to 
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Level 4 


equipment ratios of greater than one to one. I modify instructional methods to take 
advantage of the learning styles of individual students. 

I continuously try new approaches suggested by research or observation to discover 
the most effective means of using technology to engage my students and meet 
curricular goals. I work with a team of fellow teachers to create, modify and improve 
my practices in this area. 



IV. Assessment of student performance 

Level 1 I evaluate my students using objective tests only. 



Level 2 


I evaluate some student performances or projects using subjective criteria. I save 
some student work for cumulative folders and parent conferences, and print some 
electronically produced student work. 


Level 3 


I use a wide range of assessments to evaluate student projects and performances. I 
can create assessment tools like checklists, rubrics and benchmarks which help the 
student assess his own performance and allow me to objectively determine the 
quality of student work. I ask students to keep both a physical and electronic portfolio 
of their work. Students and their parents have the means to continuously access the 
recorded progress students are making toward their learning goals through networked 
grade books and portfolios. Students are given the opportunity to demonstrate skills 
through performance to a wide audience via data and video networks. I have a means 
of aggregating performance data for my class which I use to modify my teaching 
activities and strategies. 


Level 4 


I continuously try new approaches suggested by research or observation to discover 
the most effective means of using technology to help assess student learning. I work 
with a team of fellow teachers to create, modify and improve my work in this area. 



V. Individualization of the educational program 



Level 1 


I modify my curriculum or instructional methods only for students with identified 
special needs. 


Level 2 


I occasionally give students the choice of assignments in my class, but all class 
members (unless they are in special education) must meet in the same learning 
objectives within the same time frame. Skill remediation is done during summer 
school or informally during or after school. 


Level 3 


With the assistance of the student, parents and appropriate specialists, I create an 
individualized learning plan for each of my students. I track the accomplishment of 
learning goals in the plan using a computerized tool. I use this tool during parent 
conferences and for school or state reporting. Students and their parents have 
networked access to this tool for continuous monitoring of progress and plan 
modification. 


Level 4 


I provide suggestions about the content and design of the individualized 
computerized planning and report tools. 



VI. Professional growth and communication 

Level 1 I do not use electronic resources for professional growth or communication. 

Level 2 I can find lesson plans and some research in on-line data bases. I correspond with 
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parents and other teachers using e-mail. 

Level 3 I use the Internet and other on-line resources to obtain research findings, teaching 
materials and information related to the content of my classes. I read electronic 
newsletters and journals to keep current on educational practices. I participate in 
electronic discussion groups and chat rooms which are related to my area of 
education, and both contribute to and use the best practices discussed there. I use a 
computerized presentation program when giving workshops or speaking at 
conferences. I use technology to take part in distance learning opportunities for my 
own professional development. 

Level 4 I organize professional growth opportunities for other teachers and feel comfortable 
teaching other staff members about the use of technology. 



VII. Research and evaluation of technology use 

Level 1 I have not attempted to determine whether the use of instructional technology has 
made a difference in my students’ learning or classroom climate. 

Level 2 I gather, use and share anecdotal information and observations about student use of 
technology in my classroom. 

Level 3 I use action research and aggregated data to accurately determine whether the 

technology and methodology I am using has an impact on how well my students learn 



and on school climate. 

Level 4 I participate in formal studies of the impact of technology on student learning 

conducted by professional groups and academics. I have designed such studies as part 
of my own professional education. I report electronically and in print the findings of 
my research to other professionals. 



Self-Evaluation Rubrics for Teacher Internet Use 



To successfully complete this class you must be able to demonstrate mastery of each of the following 
skills. You will keep a portfolio of recorded searches, print-outs, journal entries, professional 
articles, etc. which demonstrate the competencies. This checklist will: 

1) Allow the effectiveness of my instruction to be evaluated 

2) Give you, the learner, a guide to the skills you must master to meet the course outcome. 

3) Provide an indicator of what you need to do to complete the class. 

I. Internet basics 

Level 1 I do not understand how networks work, nor can I identify any personal or professional 
uses for networks, including the Internet. I do not have an account on any network nor 
would I know how to get one. 

Level 2 I can identify some personal or professional uses for networks, and understand they have 
a value to my students and me. I’ve read some articles about the Internet in the popular 
press. I can directly use network access to a library catalog or CD-ROM. 

Level 3 I can describe what a computer network does and how it can be useful personally and 

professionally. I can distinguish between a local area network, a wide area network, and 
the Internet and can describe educational uses for each. I can describe the history of the 
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Level 4 


Internet, recognize its international character, and know to a degree the extent of its 
resources. I have personal access to the Internet that allows me to receive and send email, 
download files, and access the World Wide Web. I know that I must protect my 
password, and should restrict access by others to my account 

I use networks on a daily basis to access and communicate information. I can serve as an 
active participant in a school or organizational planning group, giving advice and 
providing information about networks. I can recommend several ways of obtaining 
Internet access to others. 



II. Email and electronic mailing lists 
Level 1 I do not use email. 



Level 2 


I understand the concept of email and can explain some administrative and educational 
uses for it. 


Level 3 


I use email regularly and can: 

• read and delete messages 

• send, forward and reply to messages to 

• create nicknames, mailing lists, and a signature file 

• send and receive attachments 


Level 4 


• use electronic mailing lists and understand the professional uses of them 

• read and contribute to a professional electronic mailing list 

I can send group mailings and feel confident that I could administer an electronic mailing 
list. I use activities that require email in my teaching. I can locate lists of subject oriented 
mailing lists. 



III. The World Wide Web 

Level 1 I do not use the World Wide Web. 



Level 2 


I am aware that the World Wide Web is a means of sharing information on the Internet. I 
can browse the Web for recreational purposes. 


Level 3 


I can use a Web browser like Explorer or Netscape to find information on the World 
Wide Web, and can list some of the Web’s unique features. I can explain the terms: 
hypertext, URL, http, and html. I can write URLs to share information locations with 
others. I can use Web search engines to locate subject specific information and can create 
bookmarks to Web sites of educational value. 


Level 4 


I can configure my web browser with a variety of helper applications. I understand what 
“cookies” do and whether to keep them enabled. I can speak to the security issues of on- 
line commerce and data privacy. 



IV. Search tools 

Level 1 I cannot locate any information on the Internet. 



Level 2 


I can occasionally locate useful information on the Internet by browsing or through 
remembered sources. 


Level 3 


I can conduct an efficient search of Internet resources using directories likeYahoo or 
search engines like Excite, Lycos, or Infoseek. I can use advanced search commands to 
specify and limited the number of hits I get. I can state some guidelines for evaluating the 
information I find on the Internet and can write a bibliogigp^ic citation for information 
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Level 



V. 

Level 

Level 

Level 



Level 



VI. 

Level 

Level 

Level 



Level 



VII. 

Level 

Level 

Level 



Level 



VIII. 

Level 

Level 

Level 



found. 

4 I can identify some specialized search tools for finding software and email addresses. I 
can speculate on future developments in on-line information searching including know- 
bots and other kinds of intelligent search agents. 

Newsgroups, gophers and telnet 

1 I have no knowledge of newsgroups, gophers, or telnet functions. 

2 I know that there are resources in a variety of formats available on the Internet, but 
cannot confidently access them. 

3 I read the newsgroups that interest me on a regular basis, and I can contribute to 
newsgroups. I understand the use of gophers and can locate several that help me. I can 
write directions to locating a gopher so that others can find it as well. I can access a 
remote computer through the telnet command, including remote library catalogs. I can 
find the help screens when emulating remote computers and can log off properly. 

4 I know how to find, configure, and use the specialized tools for newsgroups, gophers, and 
telnet access. I use the resources found in these areas with my students. 

Obtaining, decompressing, and using files 

1 I cannot retrieve files from remote computers. 

2 I know that documents and computer programs that are useful to my students and me are 
stored on computers throughout the world. I cannot retrieve these files. 

3 I understand the concept and netiquette of “anonymous FTP” sites. I can transfer files 
and programs from remote locations to my computer, and can use programs or plug-ins 
that help me do this. I can extract compressed files, and know some utilities that help me 
view graphics and play sounds and movies. I understand the nature and danger of 
computer viruses, and know how to minimize my risk of contracting a computer virus. 

4 I use information I have retrieved as a resource for and with my students. I understand 
the concept of a network server, and the functions it can serve in an organization. I can 
use an ftp client to upload files to a server. 

Real-time and push technologies 

1 I use only static documents and files I retrieve from the Internet. 

2 I have some information sent to me on a regular basis through e-mail and I check some 
sites on a regular basis for information. 

3 I use chat-rooms and customized news and information feeds. I can listen to audio 
streamed from the web. I know the hardware and software requirements for web-based 
videoconferencing. 

4 I can use real-time applications to design a “virtual” classroom or interactive learning 
experience. My students use videoconferencing for communication with experts and 
project collaboration with other students. 

Webpage construction 

1 I cannot create a page which can be viewed with a web browser. 

2 I can save text I’ve created as an html file with a command in my word processor. I know 
a few, simple html commands. 

3 Using hand-coded html or a web page authoring tool, I can: 

- view web pages as a source documents 
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- create a formatted web page that uses background color, font styles and 
alignment, graphics, and tables 

- include links to other parts of my document or other Internet sites in my page 

- know basic guidelines for good web page construction and the district’s web 
policies 

Level 4 I can use the web as an interface to databases. When appropriate, I can register my pages 

with search engine sites. I can help write web creation policies for design, content, and 
use. 

IX. Learning opportunities using the Internet 

Level 1 I am not aware of any ways the Internet can be used with students in my classroom. 

Level 2 I occasionally allow my students to use the Internet to find information. 

Level 3 I know a variety of projects and activities that effectively use the Internet to instruct and 

involve students. I know a source for collaborative projects, can direct students to on-line 
tutorials and learning resources, and encourage a variety of key-pal activities. 

Level 4 I can design and implement an Internet project or maintain an educational Internet site. 

X. Netiquette, On-line Ethics , and Current Issues Surrounding Internet Use in K-12 
Schools 

Level 1 I am not aware of any ethics or proprieties regarding the Internet nor am I unaware of any 
issues dealing with Internet use in a school setting. 

Level 2 I understand a few rules that my students and I should follow when using the Internet. I 
understand that the Internet is sometimes a controversial resource which many educators 
and parents do not understand. 

Level 3 I have read a guideline for Internet use such as Rinaldi’s “The Net: User Guidelines and 
Netiquette” or other source, and follow the rules outlined. I know and read the FAQ files 
associated with sources on the Internet. I am aware that electronic communication is a 
new communications medium that may require new sensitivities. I can identify print and 
on-line resources that speak to current Internet issues like: 

- censorship/site blocking software 

- copyright 

- legal and illegal uses 

- data privacy 

- security 

I can list some of the critical components of a good Acceptable Use Policy and know and 
use our district’s. 

Level 4 I can use my knowledge of the Internet to write good school policies and activities that 
help students develop good judgment and good information skills. 

Permission to freely print and use these rubrics is given. Complete and updated versions of these 
rubrics plus other assessment tools for staff development in technology can be found Doug Johnson's 
The Indispensable Teacher's Guide to Computer Skills: a Staff Development Guide, Linworth 
Publishing, 1998 (800-768-5017). Contact Doug Johnson at 507-387-7698 or 
johnsd9@mail.mankato.msus.edu for further information. 
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Please provide the following information: 



Name of the school 

School address 

School phone number . 



1 . What grade levels do you teach? ( please check all that applf) 

K 3 6 9 12 

1 4 7 10 Other 

2 5 8 11 



2. What subject areas do you teach? (please check all that apply) 



D 
D 

a 

□ 
□ 
)□ 

3. How do you classify your main assignment at the school? 



Regular full-time teacher □ 

Regular part-time teacher □ 

Itinerant (you teach at more than one school) □ 

Long-term substitute □ 

Other ( please specify ) □ 



Elementary education (all subjects combined) □ 



Mathematics 


□ 


Science 


Social Studies 


□ 


Language Arts 


Second Languages 


□ 


Fine Arts 


Physical Education 


□ 


Health 


Vocational education 


□ 


Computers 


Special education 


□ 


Other (please specify 
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4. As of the end of the last school year, how many years had you been teaching? 

5. How many total students do you teach each week? 

6. What is your average class size? 

7. Do you have a computer in your classroom? (If you use more than one classroom, think about the one 
you spend the most time in for this and all other questions) (please check one) 

Yes, one □ Yes, more than one □ No □ 

8. Do you have any computers in your classroom that are connected to the Internet? ( please check one) 

Yes, one □ Yes, more than one □ No □ 

9. How many hours does your average student spend on the computer at school in an average week? 

10. How many hours does your average student spend using the Internet at school in an average week? 

1 1 . Approximately how often do you use each of these applications with your students? (check one) 



Once or 



Not 



Daily 


Weekly 




Monthly 


twice a year 


Never 


Avail a 


♦ 


Computers in general 


□ 


□ 


O 


O 


O 


□ 


♦ 


Word processing packages 


□ 


□ 


O 


O 


o 


□ 


♦ 


Spreadsheets 


□ 


□ 


o 


o 


o 


□ 


♦ 


Databases 


□ 


□ 


o 


o 


o 


□ 


♦ 


Graphical applications 


□ 


□ 


• □ 


o 


o 


□ 


♦ 


Presentation software (e.g., Power Point) 


□ 


□ 


□ 


o 


o 


□ 


♦ 


Desktop publishing 


□ 


□ 


□ 


o 


o 


□ 


♦ 


Any Internet activity 


□ 


□ 


□ 


o 


o 


' □ 


♦ 


Search engines for the Internet (e.g., Infoseek, Yahoo) 


□ 


□ 


□ 


o 


o 


□ 


♦ 


Hypermedia/Multimedia (e.g., HyperCard) 


□ 


□ 


□ 


o 


o 


□ 


♦ 


Integrated Learning Systems (e.g., Jostens, CCC) 


□ 


□ 


□ 


o 


o 


□ 


♦ 


Simulation Programs 


□ 


□ 


□ 


o 


o 


□ 


♦ 


Drill/Practice Programs, Tutorials 


□ 


□ 


□ 


o 


o 


□ 
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1 2. How do students use computers in your classes? (check all that apply) 

□ to organize and store information 

□ to collect data and perform measurements 

□ to manipulate/analyze/interpret data 

□ to communicate information as the result of investigations 

□ to create visual displays of data/information (e.g., graphs, charts, maps) 

□ to plan, draft, proofread, revise, and publish written text 

□ to create graphics or visuals of non-data products (e.g., diagrams, pictures, figures) 

□ to create visual presentations 

□ to perform calculations 

□ to create models or simulations 

□ to support individualized learning 

□ for remediation for basic skills 

□ to compensate for a disability or limitation 

□ other ( please specify ) 

13. How do you use the Internet in your classes? (Check all that apply) 

□ to gather information from a variety of sources 

□ to communicate with others outside of the school 

□ other (please specify ) 

14. In an average week, you may take on a variety of roles. What percentage of the time do you think you 
act in each of the following roles: 

Lecturer % 

Coach % 

Mediator % 

Facilitator % 

T otal 100 % (should add up to 100 percent) 

15. Have you received any professional development in the use of technology during the past school year 
(1 997-98)? (please check one) 

Yes □ No □ 
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Professional Development Activities 



WHAT PROFESSIONAL DEVELOPMENT SHOULD BE INCLUDED? 

• Report all professional development related to the use of technology that you 
participated in over the past year, including the summer of 1997 and the 1997-98 school 
year. 

• Do not report professional development not related to technology (e.g., reading), but do 
report professional development in specific subject areas that incorporated technology 

applications. 





N 


Y 


# of 
Hours 


Did you lead this 
activity? 






N 


16. In the past year did you participate in or lead any of the following j 
types of professional development activities related to 
technology? We are treating these categories as mutually 
exclusive so please report hours for each activity under on 
category only. 




a. Within-district workshops or institutes, focused on a 
specific topic, provided by or within the district. 












b. Out-of-district workshops and institutes, focused on a 
specific topic, provided outside of the district. 












c. Courses for college credit. 












d. Teacher collaboratives or networks, connecting teachers 
regionally, state-wide, nationally, or internationally (do 
not include activities described in questions a through c). 












e. Out-of-district conferences, provided by professional 
organizations, regional centers, the state department of 
education, etc. 












f. Immersion or internship activities, in which a teacher 
spends a concentrated period of time working in a lab or 
industrial setting with professionals in his subject area. 












g. Receiving mentoring, coaching, lead teaching, or 
observation, in a one-on-one situation, usually in the 
classroom. 












h. Teacher resource center, which provides professional 
development materials and is staffed by a lead or 
resource teacher. 












/. Committees or task forces focusing on curriculum, 
instruction, or student assessment. 












j. Teacher study groups that meet regularly, in face-to-face 
meetings, to further your knowledge in your discipline or 
of pedagogical approaches. 












Other forms of organized professional development related to 
technology. (Do not include reading or other work you 
have done on your own.) Specify 
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17. In the past year, from the summer of 1 997 through the 1 997-98 school year, did you participate in any of 
the following types of individual professional development related to technology? 

a. Individual research project, in which you examine your own teaching and your students’ 
learning. 

□ Yes hours in the past year 

□ No 

b. Individual learning, in which you read journals or other professional publications, browse the 
Internet, etc. 

□ Yes hours in the past year 

□ No 

c. Other forms of individual professional development related to the use of technology in teaching 

(specify) 

□ Yes hours in the past year 

O No 

18. Approximately how many hours did you spend during the 1997-98 school year in all types of 

technology-related professional development activities? Include all of the types of activities listed in 
Questions 1 & 2. 

19. How much do you believe that technology has changed or determined the way you teach your classes? 
(p/ease check one) 

Greatly □ Somewhat O Not at all □ 
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20. Think of up to three of the technology-related professional development activities you participated in 
during the 1997-98 school year. Using the activities codes below, indicate the type of activity and the 
extent to which you agree with statements in the chart. 



Activities 


a. Within-district workshops or institutes. . . 


h. Teacher resource center. . . 


b. Out-of-district workshops and institutes. . . 


i. Committees or task forces. . . 


c. Courses for college credit 


j. Teacher study groups... 


d. Teacher collaboratives or networks. . . 


k. Other forms of organized professional 


e. Out-of-district conferences. . . 


development... 


f. Immersion or internship activities. . . 


1. Individual research project 


g. Receiving mentoring, coaching, lead 


m. Individual learning 


teaching, or observation... 


n. Other forms of individual professional 




development 



Key: SA - Strongly Agree, A = Agree, D = Disagree, SD - Strongly Disagree 



a. Well matched to you own goals for your 
professional development 


Activity 1 


Activity 2 


Activity 3 


SA A D SD 


SA A D SD 


SA A D SD 


b. Well matched to your school’s or 
department’s plan to change practice 


Activity 1 


Activity 2 


Activity 3 


SA A D SD 


SA A D SD 


SA A D SD 


c. Based explicitly on what you had learned 
in earlier professional development 
experiences 


Activity 1 


Activity 2 


Activity 3 


SA A D SD 


SA A D SD 


SA A D SD 


d. Followed up with activities that built upon 
what you learned in this professional 
development activity 


Activity 1 


Activity 2 


Activity 3 


SA A D SD 


SA A D SD 


SA A D SD 


e. Designed to support reform efforts 
underway in your school 


Activity 1 


Activity 2 


Activity 3 


SA A D SD 


SA A D SD 


SA A D SD 


f. Designed to support state or district 
curriculum frameworks 


Activity 1 


Activity 2 


Activity 3 


SA A D SD 


SA A D SD 


SA A D SD 


g. Designed to support state or district 
assessment 


Activity 1 


Activity 2 


Activity 3 


SA A D SD 


SA A D SD 


SA A D SD 
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21 . Rate your experience and comfort level in each of the following applications using a scale of 1-4 



Experience Comfort 



where 1 = a lot of experience 1 

2 = some experience 2 

3 = little experience 3 

4 = no experience 4 



= very comfortable, 

= moderately comfortable, 

= would need some help to feel comfortable, and 
= would need a lot of help to feel comfortable 



! 


Experience 


Comfort 

level 


Computers in general 






Word processing packages 






Spreadsheets 






Databases 




► 


Graphical applications 






Presentation software (e.g., Power Point) 






Desktop publishing 






Internet software (e.g., Netscape) 






Search engines for the Internet 






Hypermedia/Multimedia (e.g., HyperCard) 






Integrated Learning Systems (e.g., Jostens; CCC) 






Simulation Programs 






Drill/Practice Programs/Tutorials 
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Department of Educational Accountability 
MONTGOMERY COUNTY PUBLIC SCHOOLS 
Rockville, Maryland 20850 

TEACHER SURVEY: GLOBAL ACCESS 

FY 1997-1998 



How many years have you taught at this school? 



Computer Access 

Do you have a computer in your classroom? 

Do you have Netscape access from that computer? 
Do you have a First Class account? 

Do you have a computer at home? 

If vo u own a computer, answer the following : 

Does your home computer have a CD-ROM drive? 
Does your home computer have Internet access? 
Can you access First Class from home? 



Yes 


No 


Not Sure 


□ 


□ 


u 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 



Resources Used for Planning: Check the appropriate box(s) to indicate the resource(s) that you * ^ yi 
used to plan and implement your instructional program during this academic year, i- i :y s &K 



□ Curriculum resources found on First Class 

□ Curriculum resources found on the MCPS Website 

□ Curriculum resources found on other locations on the Internet 

□ On-line resources (e.g., Studyworks, Electric Library, ProQuest, etc.) 

□ Information found on an educational listserve discussion 

□ Lesson(s) that you found on the Web 



If you checked any of the resources above, please name two specific resources that you found particularly useful. 



Support: Please put a check next to those who have provided you with support in integrating technology : ■ ; 
into your work. . . V-,h ^ ' -■ j. . 

Teachers in my school 

□ School media specialist 

□ Instructional technology specialist, (OGAT) 

□ Curriculum specialist in my subject area 

□ Parent/community volunteer 

□ Students 

□ User support specialist in my school 

Level of Use: Which of the following statements best describes your use of technology in ypur 
instructional program? \ ■, • • " Vi- y, 

□ Technology is fully integrated into my instructional program. 

□ I have integrated technology into specific instructional units/projects. 

□ I use technology infrequently with students. 

□ I have not used technology at all in my instructional program. 

BEST COPY AVAILABLE 
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Training 

Check any of the training 




activities listed below that 
participated in during this 
academic vear 


you 


School based training given 
by Global Access trainer 


□ 


School based training given 
by school-based staff (tech 
committee chair, other 
technology leaders) 


□ 


Instructional Technology 
Integration Project (ITIP) 
training 


□ 


Content specific training 
(e.g., math, science, etc.) 


□ 


Countywide workshops 
during the school year 


□ 


Summer training 


□ 


A distance learning course 


□ 


Technology related 
conference 

(e.g., MICCA, Classroom 
Connect, etc.) 


□ 


College or university 
technology related course(s) 


□ 



For each training activity 
checked indicate the topics 
and the approximate 
number of hours spent. 



Indicate your level of 
satisfaction with each training 
activity checked. 



Very 

Topics Hours Satisfied Satisfied Not Satisfied 



□ □ □ 

□ □ □ 

□ □ □ 

□ □ □ 

□ □ □ 

□ □ □ 

□ □ □ 

□ □ □ ! 

' 

□ □ □ j 



Indicate approximately how often you 
engaged in this activity 



i 





Rarely 


Frequently 


Regularly | 




(Once or 


(Several times 


(Approx, once j 




twice) 


per marking 


per week or 






period) 


more) 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


□ 



TTUT7 



Administrative Tasks 

Indicate which of the following administrative 

tasks you engaged in during this academic 

year . 



Used technology to keep track of student 
progress/achievement (e.g., grades, interims) 
Used technology to keep track of attendance 
Obtained school-wide information via First Class 
Obtained district-wide administrative information 
via First Class 
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ComDuter Tools/ Applications 




ran a cirteck BiexS So any 




For every composer 




For every application/ 


computer tool/application 




tool/application you checked 


tool checked, indicate 


you used during thjs 




indicate approximately how often 


whether the teacher used 


academic vear. 




it was used. 




it directly and/or the 












students used it directly. 












Check both if applicable. 






Rarely 


Frequently 


Regularly 


Teacher used 


Students 






(Once or 


(Several times 


(Approx. 


directly 


used 






twice) 


per marking 


once per 




directly 








period) 


week) 






Word processing 


□ 


□ ' 


□ ffff 


IS □ 






Worked on/created data base 


□ 


□ 


□ 


□ 


□ 


□ 


Spreadsheet 


□ 


'? □ ' ■(: 










Desktop publishing 


□ 


□ 


□ 


□ 


□ 


□ 


Charts/graphing: . 


□ 


£ ' □ 




f-,.: □ 


(r 


iff- ED -'5'' 


Graphic organizer 


□ 


□ 


□ 


□ 


□ 


□ 


Statistical package V • . ,i ; € (f 5y 


m. 


If □ 


□ *• 




•f/PDffff 


. •; f;: 


Multimedia presentation 


□ 


□ 


□ 


□ 


□ 


□ 


"Computer programming 


fD 


■: □ • 


#: » V jy 






□ 


Simulations 


□ 


□ 


□ 


□ 


□ 


□ 


Digital camera/scanner 


□ 




vf;,: 


■•/f fDf v- 


f : :• f D; : r , : 




Digital imaging software (e.g., 


□ 


□ 


□ 


□ 


□ 


□ 


NIH Image or PC Image) 














Image manipulating software 


□ 


□f i : ; 


iff - ■ □ -f : if 

: * ■ r •*;••• ' if ; hy : 


-D -ff 


ff fppffrjf 




(e.g., pnoio onop, 1 6 ic , E 
Graphing calculator 


□ 


□ 


□ 


□ 


□ 


□ 


sProbeware 7 ?f - f f ’,,f / S 






• : :-f □ , r\ 






□ 


Content specific software or 


□ 


□ 




□ 


□ 


□ 


CD-ROMs 














On-line newspaper/magazine 


fD 


f : 


if-::-.,: (.'Pffff 




vf- 


y 4 ‘ ... y v t j. 


if ( creati ng projects iff } 




i;’> ■ ' . 


f ' ' * f '' f 




\tv i 




On-line exchanges (penpal, 


□ 


□ 


□ 


□ 


□ 


□ 


scientific info collection 














activities 














Web page development 






f. /-•••.□ flfl 


• :--f □ j',-v 


v;;rL-::v 




Video conferencing 


□ 


□ 


□ 


□ 


□ 


□ 

- n <•* 


Retrieving information from the 


□ 


yf((3f v ;( 


fff □ 




f f'.f □ . f f 


• □••y 


:• f Intemetii' f f • f ■ .-ff; 




: 


'• ■ * \ - 1‘ V. 


■ ■ i. 


; Jv'-.f ,/ ; Vk 


:> f f r - V} 


Retrieving information from 


□ 


□ 


□ 


□ 


□ 


□ 


electronic resources on the 














Research and Learning Hub 












U 


/Media Center 















Teaching Practices - 

In what ways, if any, has the use of technology changed your 
instructional practices? 

I spend less time lecturing to the whole class. . 

I spend more time with individual students. 

I am more comfortable with small group activities. 

I am more comfortable with students working independently. 

I am better able to differentiate instruction. 

I am better able to present complex material to my students 
I am better able to assess students’ work in particular subject 
areas. ' V.. .. :v. , ' , -v .. . • ... v. 
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Yes 

n 


No 


Not yet, but 

1 think 1 will 

f' pi f 


Doesn’t 

apply 

; fi* : v 


□ 


□ 


□ 


□ 


'□ V 


•• □ 


□ f 


□ 


□ 


□ 


□ 


□ 


’ □ 


□ 






□ 


□ 


□ 


□ 


□ 


□ 


• • ■ □ 


a.: 



o 

ERIC 



An Educator's Guide to Evaluating the Use of Technology in Schools and Classrooms 



Page 107 



Planning 




X ■' «. <: 


s,. . -.v 


*£*5 „ ■& i 


Rate your ability to perform the following planning 
activities involving technology. 


Highly Abie 


Able 


Not Able 


Doesn’t 


Connect instructional uses of technology to curriculum 

nhiopti\/DC onH oti 


/U;:? □ \ 




□ 


Apply 

□ 


uuj wiivtsb ciMu siuueru acnievemeni 
Identify the appropriate technology 


□ 


□ 


□ 


□ 


Accommodate for different levels of students’ 








□ 


•;i©cnnoioyy expertise 

Determine how the students’ projects will be evaluated 


□ 


' □ 


□ 


□ 


Estimate overall time required to complete projects 


u \'; -:m 






□ 


Prepare student materials (templates, student 


□ 


□ 


□ 


□ 


checklists, grading sheets) 

Determine how students will be organized, how time 
will be allocated, and how technology will be shared 




Wn ^ 


*r' 


□ 





*** : 

^\| l » & wp-**- '•* 

i* i .Jt ^ i 


A#. X5 *j* 

t, r.4x<cv<i:Lv *■ 


M 

4 i 4 !.:# Si ;■ 


Indicate whether any of the following are barriers to integrating 


Not a 


Minor 


Major 


technology into your instructional program. 


Barrier 


Barrier 


Barrier 


Not enough or limited access to computer hardware 


GH", ;• 


: iX O 


□ 


Not enough computer software 


□ 


□ 


□ 


Purchased software has not been installed ? •' -p.-.’ 


■tf. □ 


• ? ;□ 


□ 


Lack of time in school schedule for projects involving technology 


□ 


□ 


□ 


Use of technology not integrated into curriculum documents 




□ 


□ 


Lack of adequate technical support for technology projects 


□ 


□ 


□ 


Not enough teacher training opportunities for technology projects 


ZfM ' ■ 


i ! □ ■ 


□ 


Lack of knowledge about ways to integrate technology to enhance 
curriculum 


□ 


□ 


□ 


Technology integration is not a school priority 






□ 


Difficulty finding substitutes in order for teachers to attend training 


□ 


□ 


□ 


Hiring substitutes in order for teachers to attend training is discouraged 




□ 


□ • 


Students do not have access to the necessary technology at home 


□ 


□ 


□ 



We are very interested in any comments you have regarding your experience with technology in 
general or Global Access in particular. Please write you comments in the space provided below. 
Feel free to attach additional paper. 



Thank you for taking the time to complete this questionnaire. W| predate your input. 
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cggr 



' *' *4 ^ " * '! L < '• ’ V "* v * «* *• ' | ' *<1/ ' > ♦ . , \ » *^»%**> • - * >.* '- i ^g *> mj 

" ; • Use *,Na» 2ipenal.only.:;v.^ ‘;>$:iv;^v • -U*. 



E. 



_QDCD(DCD<D<S)CSK2)CD 
_000©©0©(2)<3)(£>j 

~|CD CD CD (5) O CD (D CD <D 



• Makesolid marks that fill the oval coi^Ietefy. ’■” v ; 
..♦ Erasex^ you wish to change. 

. K Malrano stray ma rks on thb lohni^v ^ 




iTiEGHNOEOGYi? US Ef ATtfS G HOG 15; ***■ 



■ The following Hems refer to your computer use AT SCHOOL Please indicate whether you have done any 
m of the following tasks within the PAST WEEK. 



. y . OS3 
S ,-->CZ3 

C gd 



iv* . ; - 
f ...» 



Sent an E-Mail Message C : '*■ V- ■.• ■ 

Read an E-Mail Message 

Posted a Bulletin Board Message ' \ • 55 V' 

Read n Bulletin Board Message 

Had. a "Real Time" Conversation {"Chatting") 

Used First ("lass 

Used Netscape : Vvv * 

Downloaded Information from the Internet 

Used a CD-ROM in the Media Center : x '=' * 

Worked on a Databa.se/Spreadsheet 

Played a Came ’ • '*'*pr< : *•*'£■£**? ' : ' 

Wrote a Program 

Used Multimedia Software (e.g., HyperCard, Astound, Persuasion) 
Used Word-processing Software ie.g., WordPerfect, Microsoft Word) 
Used a Graphing Calculator >..* 

Made a graph or drawing 

Used a Desktop Publishing Program ' ; • . >\ : • '■ - A ' ♦ > 



Sic 



iTjEfeHNOEOie.YSI E£iE$Att£ H QM m V i 






h.'-l 

r?-:| 



The following items refer to your computer use AT HOME . P/ease indicate whether you have done any of 
the following tasks within the PAST WEEK. 



Do you have a home computer * 




k : i 









mm 



Sent an E-Mail Message :>r 
Read an E-Mail Message 
Posted a' Bulletin Board Message * « * ;• ' r ; ;-v IT m * ‘ ' G 

Read a Bulletin Board Message 

Had a "Real Time" Conversation ("Chatting") : ■- >' 

Used First Class 

I . k . 

Used Netscape- vij?‘ v ;.o : ‘ *’ . ' • :•> 

Used an Internet Browser other than Netscape (e.g.. Mosaic, Microsoft Explorer) 
Used a Commercial Internet Service (e.g.. Prodigy, AOL) ■ v Vv 
Downloaded Information from the Internet 

Looked up Informatibn'on a CD-ROM / ‘ .0 <V " 

Worked on a Data base/Spreadsheet 

Wrote a Prog ra m - ■’■■* :> ' ! v * ... 

Used Multimedia Software te.g., HyperCard, Astound, Persuasion) 

Used Word-processing Software (e.g.; WordPert^,^ 

Used a Graphing Calculator 

Played a Came - ' - D': : • 

Practiced mv Typing 



S «s?, Vi- <■*!■ *&?■ 





ts, 


>■ 




. ::0 


Jy-O .... 


o 


' o ! 


>; --v : O 


v- O'-: i 


o 


o 


/: :o 


r 


o 

<WMM 


o 


o 


o ‘ 






o 


o' 






o 


o 


O 


o:^-: 


o 


o 


. o 


;:/>o 


o 


o 


Tr^O/^ 


ri -Q v :0 


o 


o 


o 


. o 


o 


o 


o 


o 


o 


o n 


o 


o 


o 


o 


- o 


I o • 


o 


o 




ir 


o 


o 


o 


o ■ 1 


o 


o 


' o 


O’ - 


o 


o 


- -o 


.o - ; 


o 


o ! 


. o 


o j 


o 


o | 






Continue on back 
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rt 

*ri- 

■ ** ■*- 






KV' 






Li 111 1 1 1 






IHpSMl 


CifeECSIAtl# HO M 1 


jfi 



For those with a home computer, please mark whether your computer (or at least 
thereof morefha^ on^hasthe f^Jlowing^^ 

Apple/Mac intosh- : r;‘V. .* \ '• !.,<’■ *P- 

IBM/IBM Compatible 




A Printer 

A CD-ROM Drive ■ ; ‘ ‘ 

Windows 3.11 or Windows 95 

Internet Access (through any type. of provider) */,''■ ■ W 

Commercial Internet Network Provider (e.g., AOL, Prodigy) 
Ability to send/receive E-mail' - 

Ability to view Webpages (e.g., Netscape, Microsoft Explorer) 



Yes* 




y Danjlif' 
•K^OAvfc 


r'O 


crjo : 7J 


Vi-Q;:--' 


O 


o 


o 


Y'}o 




,* •v.t- 


,o 






o 


O 


6 


*:SP 


o : ; 




o 


o 


o 




:.r, ;?©.*>£ 


H 


o 


o 


o 


o •" 




u.V.o ; r;r: 


o 


o 


o 



^GENSIAL ilNFORMATHON 



[i i d:f^ O iT4^ I»A S-Sl^ 






Please indicate at which of the following 






times you used a computer LAST 






WEEK at school 












i Before School . ■ - \ • : ; ;; ‘ 




r-ob-. 


, During the Lunch Period 


O 


o 


l During a Study Period 


o 


'bo. 


After School 


o 


o 



* your grade level? 

’• ' O 6th O 7th v A- O 8th 
1 th O 1 2th 



What is your gender? 

O Male O Female 



O 9th 



O 10th 



* I 



Do you think that there are enough computers at this school for 
students who need to use them? 

O Ves • " O -V, ’ 

Last week at this school, how many hours did you spend 
using a computer? 

O I didn't use a computer O Less than 1 hour 

CD 1 to 3 hours ■ • ’ . r u O y 4 to 6 hours 

O I spent more than 6 hours 

At this school, I find computers make school work 
more interesting. 

O O O O O 

Strongly Agree Neutral Disagree Strongly 

Agree ’ . Disagree' 

At this school, using computers helps me do better on 
my school work. 

■ o o o . 

Strongly Agree Neutral Disagree Strongly 

Agree; / Disagree 

What b your approximate grade average this year? 

O About A O About B O About C 

O About D O Below D 

How much schooling do you think you will 
complete? 

O No HS Diploma O High School Diploma 

O Trade School Degree O College Degree 
O Masters/Doctorate Degree : *• >•. <\k s .. 



What is your race/ethnicity? 

O African-American O American Indian v ; . 

O Asian-American O Hispanic 

'.Or White O Other 

si? How would you rate y ou r computer at>il tties? v U ' ^ v 

Extremely Below.' \:.vAverage ^ ; Above Extremely 

Poor Average v- Average Good 

Are you i n any of thesecl asses or programs? ,, ^ 

MARK ALL ANSWER CH THATAPPL Y 

O Gifted/Honors O ESOL 

O Technology/VocationaJ O Special Education - , , *»'■ 
O Advanced Placement 



What is your zip code? 





CD CD CD CD CD 
CD CD CD CD GD 
GD CD CD CD CD 
CDCDCDCDG) 
CD CD CD. CD CD 
CD CD CD CD GD 
CD QD CD CD 
CD CD CD CD CD 
CD CD CD CD CD 
CD CD CD CD CD 



* 



CD 

O 



10 



o 

CZD 



I 
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